Harmony University Thesis

Eng.D. Medical Devices and Health Technologies

Presentation

01 December 2023

Christos TachtatzisMadeleine Grealy

Thomas H. Nown

Creating a Real-Time Movement Sonification System for Hemiparetic Upper Limb Rehabilitation for Survivors of Stroke

Background

1.

Mapping Review

2.

Movement Sonification System Development

3.

Agency Study

4.

Index

[1]: Patel et al., Stroke Association, London, 2017.[2]: Nakayama et al., Archives of Physical Medicine and Rehabilitation, vol. 75, 1994. doi: 10.1016/0003-9993(94)90161-9[3]: James Lind Alliance, 2021, https://www.jla.nihr.ac.uk/priority-setting-partnerships/stroke/stroke-rehabilitation-and-long-term-care-top-10-priorities.htm

• Cost of care to support stroke survivors with disability is expected to rise to £43 billion in 2025 [1].
• Upper limb impairment is a common disability from stroke, affecting 40% of stroke survivors [2].
• Upper limb rehabilitation towards independent living is highly sought after [3].
• Conventional methods produce limited improvements.
• Methods involving audio-based techniques show promise.

Stroke is progressing from a mortal disease to one that causes disability.

Background

Standardised methodology incorporating strategic placing of musical instruments to provide audio feedback on movements.

Music Supported Therapy

Method to create audio feedback based on captured movement data, requires a system to capture movement, obtain movement metrics of interest, create an auditory display, and emit audio.

Movement Sonification

Metronomic audio stimuli, useful for cyclic movement practice. Creates temporal templates for performers to synchronise their movements to.

Rhythmic Auditory Cueing

System

Auditory Techniques

• Completed study to investigate whether people obtain a sense of agency whilst using the KinectSon system.
• Study used a novel methodology involving pitch distortion.
• 10 participants without arm impairment and 12/14 stroke survivors reported agency with the system.

Agency Study

• Created two movement sonification systems
• Soniccup - contains a novel position estimation algorithm for position sonification from accelerometer data
• KinectSon - a real-time movement sonification system using the Azure Kinect camera

Movement Sonification System Development

• Reviewed real-time movement sonification systems used in the literature.
• Identified relationships between physical dimensions and audio dimensions for movement sonification.
• Identified relationships between anatomy and motion capture technology used in the movement sonification systems used.

Mapping Review

Contributions

Movement Sonification systems are not available commercially. System creation is required. Relevant reviews do not focus on system composition, providing motivation to review created systems to identify potential options.

A systematic search revealed 145 projects involving real-time movement sonification systems, each system was keyword-classified to converge a highly heterogeneous area of literature.

Challenges

Mapping Review of Real-Time Movement Sonification Systems

Mapping Review

Systematic Search

Figure 4

Figure 3

Figure 2

Mapping Review

Head

Wrist

Foot

Trunk

Hand

Technology-Anatomy

KinectSon

Soniccup

Movement Sonification Systems

KinectSon shows good performance for all movements.

Soniccup shows good performance for normal and fast speed, but not for slow speed.

Movement 1 = normal speed. Movement 2 = slow speed.Movement 3 = fast speed.

To evaluate the output of these systems, position similarity was compared to a Vicon Nexus system, at three different reaching speeds.

Comparison Study

Comparison Study

A question remains as to whether the KinectSon system could be used to provide meaningful feedback on movement. A method to assess sense of agency (control) was created.

Four conditions were setup. The baseline condition used position to pitch, and unmuted sound on movement. Three other conditions added a pitch distortion component to the audio. Trials contained 10 of each condition, randomised.

Agency Study

Sonification Conditions

Agency Study

Sonification Conditions

Results with non-impaired participants

Results with impaired participants

Agency Results

In Progress - Journal Article

T. H. Nown, M. A. Grealy, I. Andonovic, A. Kerr, and C. Tachtatzis, “A novel position estimation method for online movement sonification, the Soniccup”, Sensors

T. H. Nown, M. A. Grealy, A. Kerr, I. Andonovic, and C. Tachtatzis, “Assessing agency of a real-time movement sonification system through audio pitch modulation”, Journal of NeuroEngineering and Rehabilitation

Published - Journal Article

T. H. Nown, P. Upadhyay, A. Kerr, I. Andonovic, C. Tachtatzis, and M. A. Grealy, “A Mapping Review of Real-Time Movement Sonification Systems for Movement Rehabilitation,” IEEE Reviews in Biomedical Engineering, vol. 16, pp. 672–686, 2023. doi:10.1109/RBME.2022.3187840

Submitted - Journal Article

T. H. Nown, A. Kerr, I. Andonovic, M. A. Grealy, and C. Tachtatzis, "Developing a real-time movement sonification system for upper-limb rehabilitation for stroke survivors", 15 Jul 2022, 44th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Piscataway, NJ: IEEE

Conference Proceedings

Output

Project Output

• Completed study to investigate whether people obtain a sense of agency whilst using the KinectSon system.
• Study used a novel methodology involving pitch distortion.
• 10 participants without arm impairment and 12/14 stroke survivors reported agency with the system.

Agency Study

• Created two movement sonification systems
• Soniccup - contains a novel position estimation algorithm for position sonification from accelerometer data.
• KinectSon - a real-time movement sonification system using the Azure Kinect camera.

Movement Sonification System Development

Index

• Reviewed real-time movement sonification systems used in the literature.
• Identified relationships between physical dimensions and audio dimensions for movement sonification.
• Identified relationships between anatomy and motion capture technology used in the movement sonification systems used.

Mapping Review

Contributions