PISA 2022

Keith Dryburgh keith.dryburgh@gov.scot

What can the PISA assessments tell us about Scottish Education?

The happiest country? Pupils in Finland had an average life satisfacton rating of 7.41

The friendliest school system? Austrians most likely to say they make friends easily and feel they belong at their school

High fliers? Singapore achieved the highest scores for Maths, Reading and Science in PISA 2022

Already peaked? Internationally, attainment in PISA peaked in Maths and Science in 2009, and Reading in 2012

What's in a name? Pupils from Scotland and Nova Scotia had the same score for Reading in PISA 2022

Did you know that... Fun PISA facts

What is PISA? PISA 2022 key findings - results by year - international comparisons - focus on maths - student questionnaire PISA data and research

What we'll cover today

What is PISA?

The Programme for International Student Assessment (PISA)

PISA is an assessment of 15 year olds' skills which are necessary to participate in society. It takes place on a three year cycle. Unlike other international assessments which have a curriculum approach, the PISA study has test content that is independent of the participating countries’ school curricula, with a focus upon assessing whether 15-year-olds are able to apply what they have learned in school in real life situations. PISA assesses performance in Mathematics, Reading and Science PISA started in 2000, with the 2022 assessments the eighth cycle. Scotland has participated in all cycles Pupils also undertake a background questionnaire about themselves; their attitudes, dispositions, and beliefs; their home life; and their school and learning experiences

PISA was developed and is led by the OECD, and is overseen by the PISA Governing Board. Each participating country appoints a National Project Manager to undertake the national assessments. In Scotland, the National Foundation for Educational Research is commissioned to undertake the assessments Students undertake the PISA assessment on computers at school in a three hour session - two hours of assessments; 30m for the student questionnaire; and 30m for breaks Test items are a mixture of multiple-choice questions and questions requiring students to constructy their own response. The items are organised into groups based on a passage of text describing a real-life situation More than 15 hours of test items for mathematics, reading and science are covered, with students taking different combinations of test items 94% of students recieved test forms covering 60 minutes of mathematics as the major domain, and another 60 minutes of one of the three minor or innovative domains

PISA assessments

There are 82 PISA technical standards - but these are the three that the OECD focus on:

Standard 1.11 The final weighted school response rate is at least 85% of sampled eligible and non-excluded schools. If a response rate is below 85% then an acceptable response rate can still be achieved through agreed upon use of replacement schools. Standard 1.12 The final weighted student response rate is at least 80% of all sampled students across responding schools.

Standard 1.7 The PISA Defined Target Population covers 95% or more of the PISA Desired Target Population. That is, school-level exclusions and within-school exclusions combined do not exceed 5%

PISA technical standards

40 students randomly sampled from each school
Guidance on pupils that can be excluded from sample due to SEN or language

Students

121 schools in main sample
115 first replacement
109 second replacement

Funding type (publically funded, independent)
School attainment (National qualifications)
Gender (male/mixed/female)
Area type (urban/rural/etc.

Sample sizes and targets

Stratification variables

PISA sampling

Main domain was mathematics
Innovative domain was creative thinking

Originally scheduled for 2021, but was postponed across all countries for 12 months due to the pandemic

Administration of the assessment took place at the tailend of the pandemic, which impacted on participation

For pupils in Scotland - and in many other countries - learning took place remotely for at around 8 months in the 2.5 years prior to the assessments

PISA 2022 was an exceptional assessment

PISA 2022

3,257 pupils took part in Scotland

117 schools in Scotland participated

81 countries, including 690,000 students, took part in PISA 2022

Results are published internationally and nationally on the first Tuesday in December the year following the assessment - on 3 Dec 2023 for the PISA 2022 results From this point, all international data from the assessments are available Initial focus is very much on the scores in maths, reading and science. This has led to the phenomonon of the 'PISA Shock' OECD, governments and researchers then use the data for secondary analysis

How are results published/used?

Pause! Any questions / reflections?

Key Findings

PISA 2022

Scottish sample

Similar to PISA 2018Similar to the OECD average

Lower than PISA 2018Higher than the OECD average

70%

Scotland PISA scores by year

Science

483

Reading

493

Lower than PISA 2018Similar to the OECD average

Maths

471

PISA 2022 Scores

Wales

70%

Similar for Maths Lower than Scotland for Maths and Science

Similar for maths, reading and science

Northern Ireland

Similar for Reading Higher than Scotland for Maths and Science

England

PISA 2022 UK comparisons

Relatively unusual in the data for having a significantly higher score for Reading than for Maths and Science

Scotland is similar to Northern Ireland for all domains

Scotland is similar to France, Germany and Italy for Maths and Science; but, higher for reading

Wha's like us?

Focus on maths

Peformance among boys was higher than among girls in maths (478 vs. 463) - however, girls scored higher in reading and performance in science was similar

Scotland's standard deviation in maths was 94 points, which means there was more deviation in scores than in 2015, 2012 and 2006

Maths in more detail

In PISA, a student’s socio-economic status is measured by the PISA index of economic, social and cultural status (ESCS). The higher the value of ESCS, the higher the socio-economic status. ESCS is a composite score that combines into a single score information from three components: parents’ highest level of education; parents’ highest occupational status; and home possessions, which is a proxy for family wealth). Information about these three components for each student was collected through the student questionnaire, a survey that students answered after completing the PISA cognitive assessment.

ESCS

PISA 2022 European comparisons

--21

-23

-22

-21

-25

-8

-15

-27

-33

-26

-20

-13

-36

Change in PISA maths scores between 2018 and 2022

Student questionnaire

PISA assesses a student to be 'satisfied' with their life if they report between 7 and 10 on the life satisfaction scale. In Scotland, this was 55.9 per cent of students, compared with 61.4 per cent on average across the OECD.

56%

Life satisfaction

compared to the OECD average of 55%

67%

Two thirds of students agreed that they felt that they belonged at their school - compared to 65% in 2018

Sense of Belonging

...or learners felt that their teachers were well-prepared to teach remotely

60%

...of learners felt they learnt less at home than they would have done at school

77%

...of learners felt well prepared to learn on their own

45%

PISA 2022 Experience of school building closures

This higher than the OECD average (66.1%)

...of learners in Scotland reported that their parents/carers asked how well they were doing at school at least once per week

70.3%

This was higher than the OECD average (30.4%)

...of learners in Scotland are distracted by using digital devices in most lessons

31.3%

This was higher than the OECD average (2.0 hours)

Time spent on digital devices on learning activities at school

2.2 hours

This was lower than in PISA 2018 (31.7%), and lower than the OECD average (30.4%)

...of learners in Scotland said that students didn't listen to what the teacher says in most lessons

26.9%

This was fewer than in PISA 2018 (11.8%), but higher than the OECD average (9.4%)

...of learners in Scotland were termed as 'frequently bullied'

10.6%

Student questionnaire

Covers a huge range of topics on homelife, friends, attitudes to learning, personal beliefs, bullying, relationships, etc.

Pause! Any questions / reflections?

PISA data and secondary analysis

https://www.oecd.org/pisa/data/2022database/

Maths score - highest parent education

Maths score - use of digital for leisure

PISA Data Explorer examples

Comparative studies - what can we learn from the differences between countries?

Do PISA data compliment other educational data sets?

Past PISA assessments - previous participants are now aged around 21 (PISA 2018) and 24 (PISA 2015). What more do we know about these cohorts?

What are the links between experiences of learning at home and attainment/wellbeing?

Social background - who are the 'resilient' students and what are factors linked to their attainment?