PISA 2022

What can the PISA assessments tell us about Scottish Education?

Keith Dryburghkeith.dryburgh@gov.scot

Did you know that...Fun PISA facts

What's in a name? Pupils from Scotland and Nova Scotia had the same score for Reading in PISA 2022

Already peaked? Internationally, attainment in PISA peaked in Maths and Science in 2009, and Reading in 2012

High fliers? Singapore achieved the highest scores for Maths, Reading and Science in PISA 2022

The friendliest school system? Austrians most likely to say they make friends easily and feel they belong at their school

The happiest country? Pupils in Finland had an average life satisfacton rating of 7.41

What we'll cover today

What is PISA?PISA 2022 key findings- results by year- international comparisons- focus on maths- student questionnairePISA data and research

What is PISA?

PISA is an assessment of 15 year olds' skills which are necessary to participate in society. It takes place on a three year cycle. Unlike other international assessments which have a curriculum approach, the PISA study has test content that is independent of the participating countries’ school curricula, with a focus upon assessing whether 15-year-olds are able to apply what they have learned in school in real life situations.PISA assesses performance in Mathematics, Reading and SciencePISA started in 2000, with the 2022 assessments the eighth cycle. Scotland has participated in all cyclesPupils also undertake a background questionnaire about themselves; their attitudes, dispositions, and beliefs; their home life; and their school and learning experiences

The Programme for International Student Assessment (PISA)

PISA assessments

PISA was developed and is led by the OECD, and is overseen by the PISA Governing Board. Each participating country appoints a National Project Manager to undertake the national assessments. In Scotland, the National Foundation for Educational Research is commissioned to undertake the assessmentsStudents undertake the PISA assessment on computers at school in a three hour session - two hours of assessments; 30m for the student questionnaire; and 30m for breaksTest items are a mixture of multiple-choice questions and questions requiring students to constructy their own response. The items are organised into groups based on a passage of text describing a real-life situationMore than 15 hours of test items for mathematics, reading and science are covered, with students taking different combinations of test items94% of students recieved test forms covering 60 minutes of mathematics as the major domain, and another 60 minutes of one of the three minor or innovative domains

PISA sampling

Stratification variables

Sample sizes and targets

Funding type (publically funded, independent)
School attainment (National qualifications)
Gender (male/mixed/female)
Area type (urban/rural/etc.

121 schools in main sample
115 first replacement
109 second replacement

Students

40 students randomly sampled from each school
Guidance on pupils that can be excluded from sample due to SEN or language

PISA technical standards

Standard 1.7 The PISA Defined Target Population covers 95% or more of the PISA Desired Target Population. That is, school-level exclusions and within-school exclusions combined do not exceed 5%

Standard 1.11 The final weighted school response rate is at least 85% of sampled eligible and non-excluded schools. If a response rate is below 85% then an acceptable response rate can still be achieved through agreed upon use of replacement schools. Standard 1.12 The final weighted student response rate is at least 80% of all sampled students across responding schools.

There are 82 PISA technical standards - but these are the three that the OECD focus on:

81 countries, including 690,000 students, took part in PISA 2022

117 schools in Scotland participated

3,257 pupils took part in Scotland

PISA 2022

PISA 2022 was an exceptional assessment

Originally scheduled for 2021, but was postponed across all countries for 12 months due to the pandemic

Administration of the assessment took place at the tailend of the pandemic, which impacted on participation

For pupils in Scotland - and in many other countries - learning took place remotely for at around 8 months in the 2.5 years prior to the assessments

Main domain was mathematics
Innovative domain was creative thinking

How are results published/used?

Results are published internationally and nationally on the first Tuesday in December the year following the assessment - on 3 Dec 2023 for the PISA 2022 resultsFrom this point, all international data from the assessments are availableInitial focus is very much on the scores in maths, reading and science. This has led to the phenomonon of the 'PISA Shock'OECD, governments and researchers then use the data for secondary analysis

Pause!Any questions / reflections?

PISA 2022

Key Findings

Scottish sample

PISA 2022 Scores

471

Maths

Lower than PISA 2018Similar to the OECD average

493

Reading

483

Science

Scotland PISA scores by year

70%

Lower than PISA 2018Higher than the OECD average

Similar to PISA 2018Similar to the OECD average

PISA 2022 UK comparisons

England

Similar for ReadingHigher than Scotland for Maths and Science

Northern Ireland

Similar for maths, reading and science

Similar for MathsLower than Scotland for Maths and Science

70%

Wales

Wha's like us?

Scotland is similar to France, Germany and Italy for Maths and Science; but, higher for reading

Scotland is similar to Northern Ireland for all domains

Relatively unusual in the data for having a significantly higher score for Reading than for Maths and Science

Focus on maths

Maths in more detail

Scotland's standard deviation in maths was 94 points, which means there was more deviation in scores than in 2015, 2012 and 2006

Peformance among boys was higher than among girls in maths (478 vs. 463) - however, girls scored higher in reading and performance in science was similar

ESCS

In PISA, a student’s socio-economic status is measured by the PISA index of economic, social and cultural status (ESCS). The higher the value of ESCS, the higher the socio-economic status. ESCS is a composite score that combines into a single score information from three components: parents’ highest level of education; parents’ highest occupational status; and home possessions, which is a proxy for family wealth). Information about these three components for each student was collected through the student questionnaire, a survey that students answered after completing the PISA cognitive assessment.

Change in PISA maths scores between 2018 and 2022

-36

-13

-20

-26

-33

-27

-15

-8

-25

-21

-22

-23

--21

PISA 2022 European comparisons

Lorem ipsum dolor sit amet, consectetuer adipiscing elit.

Student questionnaire

PISA 2022 Experience of school building closures

45%

...of learners felt well prepared to learn on their own

77%

...of learners felt they learnt less at home than they would have done at school

60%

...or learners felt that their teachers were well-prepared to teach remotely

Sense of Belonging

Two thirds of students agreed that they felt that they belonged at their school - compared to 65% in 2018

67%

compared to the OECD average of 55%

Life satisfaction

56%

PISA assesses a student to be 'satisfied' with their life if they report between 7 and 10 on the life satisfaction scale. In Scotland, this was 55.9 per cent of students, compared with 61.4 per cent on average across the OECD.

Covers a huge range of topics on homelife, friends, attitudes to learning, personal beliefs, bullying, relationships, etc.

Student questionnaire

10.6%

...of learners in Scotland were termed as 'frequently bullied'

This was fewer than in PISA 2018 (11.8%), but higher than the OECD average (9.4%)

26.9%

...of learners in Scotland said that students didn't listen to what the teacher says in most lessons

This was lower than in PISA 2018 (31.7%), and lower than the OECD average (30.4%)

2.2 hours

Time spent on digital devices on learning activities at school

This was higher than the OECD average (2.0 hours)

31.3%

...of learners in Scotland are distracted by using digital devices in most lessons

This was higher than the OECD average (30.4%)

70.3%

...of learners in Scotland reported that their parents/carers asked how well they were doing at school at least once per week

This higher than the OECD average (66.1%)

Pause!Any questions / reflections?

PISA data and secondary analysis

https://www.oecd.org/pisa/data/2022database/

PISA Data Explorer examples

Maths score - use of digital for leisure

Maths score - highest parent education

Areas of Research Interest

Mathematics - what are the factors that support attainment?

Social background - who are the 'resilient' students and what are factors linked to their attainment?

What are the links between experiences of learning at home and attainment/wellbeing?

Past PISA assessments - previous participants are now aged around 21 (PISA 2018) and 24 (PISA 2015). What more do we know about these cohorts?

Do PISA data compliment other educational data sets?

Comparative studies - what can we learn from the differences between countries?

Generate experiences with your content.
It’s got the Wow effect. Very Wow.
Make sure your audience remembers the message.
Activate and surprise your audience.