1b. Healthcare and Well-being

Scope and Introduction

Virtual worlds (VW) are rapidly emerging as a transformative force in healthcare, offering unprecedented opportunities to revolutionise patient care, medical education, and public health initiatives. These immersive, interactive digital environments provide a dynamic platform for a wide array of applications, from sophisticated medical training simulations and advanced therapeutic interventions to engaging patient education tools. The integration of virtual models, often referred to as Digital Twins (DT), enables highly personalised treatment plans and predictive analytics, allowing for a proactive approach to health management. Beyond clinical applications, VW are also instrumental in enhancing well-being and fostering social inclusion, offering innovative solutions for mental health support, combating social isolation, and improving accessibility for individuals with diverse needs. This chapter delves into specific research topics, outlining their importance, defining existing challenges, and proposing key objectives for future research and innovation within this evolving landscape.

The application of VW in healthcare and social inclusion spans prevention, advanced diagnostics, personalised care, surgical assistance, rehabilitation, and the broader aspects of well-being and social integration. Each of these areas presents unique challenges and opportunities for leveraging immersive technologies to improve health outcomes and quality of life. The strategic importance of these applications lies in their potential to democratise access to healthcare services, provide cost-effective interventions, and offer highly personalised and engaging experiences for patients and healthcare professionals alike.

Use Cases

This section outlines key research areas within the healthcare and well-being domain, detailing their significance, current challenges, and proposed objectives for future innovation.

Prevention

1b.1 Reduction of disorders and syndromes

VW offer an accessible and cost-effective method for administering preventive programmes for several disorders (i.e. musculoskeletal, dietary, ...) or syndromes (i.e. pain) with gamification elements potentially boosting user motivation to engage in training and daily practice. This approach facilitates widespread adoption and can help reduce the incidence of related diseases. For example, keeping an active life will prevent musculoskeletal pain (i.e. back pain) as well as nutrition awareness will prevent metabolic syndromes (i.e. bulimia, anorexia, obesity).

Challenges and opportunities: Sedentary lifestyles and eating disorders often lead to sever diseases. Current research on prevention is bringing the attention on awareness and engagement of the target population. VW are suitable tools for providing preventative strategies, participation and engagement.

Research and Innovation Objectives: Develop ad-hoc applications with scalable difficulty levels that promote comprehensive and focused awareness and habits, tailored to user age and characteristics for long term engagement (2c). Ensure the technology is widely available and affordable (3a, 3b, 3c) and create a user-friendly and motivating applications (2b).

1b.2 Prevention of accidents (while driving, working, at home, ...)

Continuous, passive health monitoring of users in their living environments or while using any mean of transportation or work through the use physiological sensors (better if contactless) supports personalised nudges, enhancing safety, well-being, and contextual health assessment. This system complements existing monitoring systems (i.e. driving assistant) and support the monitoring and the pre alert for possible dangerous situation for the people providing augmented feedback (visual, sonorous, haptic).

Challenges and opportunities: An augmented world connected with environment (or transportation means) as well as with contactless (or non-invasive wearable) physiological sensors would provide a better comprehension of the people (general health, attention, stress, tiredness, cognitive load, ...) and it can provide more extensive feedback preventing accidents. While regulatory interest is growing, adoption remains low due to technological feasibility, interoperability and data privacy concerns.

Research and Innovation Objectives: Develop an augmented VW collecting information, data and knowledge from contactless (on non-invasive wearable) physiological sensors and from the environment (or means of transportation) able to provide a holistic feedback system (2a, 2b, 2d, 2f). Develop AI algorithms for inferring heart rate, heart rate variability, respiration rate, and stress levels (2f). Create intuitive feedback interfaces, including head-up solution, AR, and mobile applications (2b).

Early disease detection

1b.3 Age-related or progressive diseases

VW facilitate screening for several age-related and progressive diseases (i.e. cognitive decline) without the need for a continuous specialised clinical personnel supervision, enabling widespread and cost effective deployment in local communities and volunteer associations. This approach provides a more naturalistic and ecologically valid environment for screening, allowing for the collection of detailed data such as error rates, accuracy, efficacy, and behavioural outputs, thereby enabling early detection and intervention to slow disease progression.

Challenges and opportunities: Current age-related and progressive decline (like the cognitive one) often identifies pathology only when it significantly impacts daily activities. The ability to perform occasional screenings (e.g., bi-annually after age 50, annually after 65) could enable earlier detection. However, large-scale VW applications for general population screening are not yet widely adopted.

Research and Innovation Objectives: Develop an augmented VW collecting information, data and knowledge from user of any age and able to provide a holistic feedback system (2a, 2b, 2d, 2f). Develop AI algorithms for inferring possible diseases in advance (2f).

1b.4 Early prediction of diseases

Integration of VW, Digital Twins (DT) of people, combined with continuous biosensor (in general wearable physiological sensors), can predict the development of diseases (e.g. heart failure). The VW offers individuals an advanced and easily accessible feedback and warning system highlighting the potential high risk, allowing for lifestyle adjustments to prevent disease progression, and provides a more accurate and continuous information flow for the doctors. The full environment can also be beneficial for athletes experiencing cardiac stress.

Challenges and opportunities: Individuals often lack awareness of their health and risk in the absence of symptoms. Currently, the data provided by wearable devices are often not very intelligible and uncomprehensible to the individuals. Providing an augmented VW connected with own DT can works as early detection system for the medical doctors and for creating awareness and change-life-style driver for the people.

Research and Innovation Objectives: Develop specific monitoring technics that can integrate DT and wearable sensors able to provide a continuous surveillance on users' parameters (2e, 2d). Advance data analysis and risk prediction modelling (2f). Implement augmented warning systems for changes in physiological parameters (i.e. linked to future cardiac disease or heart failure) (2a, 2b).

Personalised Care

1b.5 Personalised treatment through patient Digital Twins

Developing accurate patient-specific DT using advanced system identification and modelling techniques can assist clinicians in predicting disease progression, personalising treatments, and optimising healthcare outcomes. This allows for virtual testing and refinement of different treatments, leading to optimised therapeutic strategies, reduced uncertainty, minimised side effects, and improved patient engagement in their care decisions. A specific case can be represented using the DT for virtual inspecting the specific patient body district better understanding the extent of the illness and simulate the effects of various therapeutic strategies.

Challenges and opportunities: Current illness treatment planning heavily relies on general clinical guidelines, population-level statistical outcomes, and limited patient-specific data, leading to clinical uncertainty, trial-and-error methods, and non-optimal therapeutic outcomes. Existing patient-specific modelling methods often lack accuracy, adaptability, and scalability for clinical practice. Gaps exist in utilising advanced identification methods, augmented sensing, and integrating physics-inspired data blocks crucial for patient-specific DT development in healthcare. Moreover, a specific “care” companion can support in understanding the care and providing support on care-adherence.

Research and Innovation Objectives: Develop specific patient DT integrate in a full personal care VW connected to a network of sensors wearable and in the living environments (2a, 2d, 2e, 2f). Manage the personalised data in a security system providing privacy while maintain the whole medical chain informed and updated on patient condition (2d, 3b). Need of a specific interfaces for clinical staff and eventually a customised one for explaining the care to the patient (2b).

1b.6 Adaptation of Virtual Worlds according to patient needs (AI and DT-Supported)

Digital twins, as virtual replicas of patients, are transforming personalised healthcare by enabling real time monitoring, predictive modelling, and tailored treatment plans. Increased computational power facilitates the merging of various domains to address all aspects of a patient's health and clinical issues. Moreover, XR technologies can enhance personalisation, emotional engagement, and accessibility for all users and specifically for elderly individuals with cognitive impairment or children/adults with special needs. This approach has the potential to reduce reliance on sedatives and improve overall therapeutic outcomes also improving the therapy adherence, while promoting emotional well-being and (cognitive) engagement.

Challenges and opportunities: Existing patient DT examples often focus on specific topics and lack interoperability for clinical specialists from diverse backgrounds. The integration of DTs into routine clinical practice is still in its early stages. For example, elderly individuals often receive sedatives to manage discomfort or participate in traditional group-based reminiscence therapy, which is often less personalised and engaging. Conventional methods face limitations including reduced accessibility, limited individualisation, dependence on facilitator skills, low sensory stimulation, and difficulties for participants with cognitive or communication impairments. While solutions exist in the market, they are not widely adopted as a standard of care.

Research and Innovation Objectives: Develop specific patient DT integrate in a full personal care VW connected to a network of sensors wearable and in the living environments (2a, 2d, 2e, 2f). Manage the personalised data in a security system providing privacy while maintain the whole medical chain informed and updated on patient condition (2d, 3b). Need of specific and different interfaces for individuals and clinical staff (2b).

Surgery

1b.7 Real-time intraoperative Digital Twin for surgical assistance

This system receives real-time sensor data during surgery and continuously predicts tissue response, suggesting optimal surgical paths. It acts as a closed-loop assistant, providing warnings and suggesting adjustments, and can be visualised in XR to give surgeons intuitive feedback, ultimately enhancing surgical safety and precision.

Challenges and opportunities: Current surgical workflows primarily rely on static pre-operative imaging and intraoperative visual interpretation. While some robotic and image-guided tools offer updated navigation, they lack dynamic, patient-specific physiological models that adapt to surgical events. Real time feedback or predictive control based on continuously updating DTs remains an early research stage, with most intraoperative decisions still heavily reliant on surgeon experience. Gaps exist in using real time identification methods, developing hybrid models, fusing sensory data, and creating predictive control loops.

Research and Innovation Objectives: Develop hybrid and data-driven modelling techniques for real time surgical support (2d, 3b). Advance system identification and dynamic modelling of soft tissues and implement real-time control loops, model updates, and feedback mechanisms with new sensor fusion capabilities (2a, 2e, 2f). Create XR visualisation with low-latency interaction for surgical environments provided by a safety-critical systems with fail-safe modes for surgical applications (2b, 2c).

1b.8 Endoscopic inspection as a diagnostic procedure

Enhancing information available to physicians through advanced image sensors and multi-modality imaging systems, potentially combined with smart pill cameras, paves the way for patient-specific treatment planning. The integration with XR can make procedures faster, more accurate, and with fewer side effects, by improving surgical and endoscopic outcomes, providing intraoperative guidance, and monitoring smart pill passage.

Challenges and opportunities: Challenges include the miniaturisation of imaging sensors, the integration of multi-modal data facilitated AI, and the provision of AI-based decision support for diagnostics and treatment planning. Embedding real-time information during diagnostic and therapeutic interventions and combining it with previously collected data (e.g., live endoscopic imaging with smart pill images) remain key hurdles. While endoscopic procedures are widely adopted, these enhancements represent significant improvements within existing practices.

Research and Innovation Objectives: Develop miniaturised imaging sensors, including hyperspectral imaging, to provide high-quality imaging and video at relevant wavelengths and integrate multi-modal sensor data fusion for enhanced decision support (2a). Create XR-enhanced, intuitive visualisations and interfaces for surgeons and physicians (2b, 2c). Develop AI models and DTs for training, learning, and surgery planning (2f).

Rehabilitation – Therapy Monitoring

1b.9 Extended Reality technology to support rehabilitation intervention

Personalised and adaptive XR environments can support rehabilitative interventions, even without constant therapist supervision. This allows for longer training periods, potentially in a combined modality where XR acts as an add-on to standard therapy, enabling more patients to receive intervention simultaneously, eventually at home. Multi-user environments could further enhance adherence and motivation.

Challenges and opportunities: Currently, therapists typically provide one-to-one sessions, often limited by strict patient schedules and time constraints and forcing the therapist to be present throughout the session. While some XR solutions are applied in research hospitals, they are not widely adopted, and patients often receive a limited number of scheduled sessions before discharge.

Research and Innovation Objectives: Develop ad-hoc designed applications specifically for rehabilitation (2b). Create lightweight, wearable hardware devices (2a). Design user-friendly interfaces (2c). Implement adaptive mechanisms to adjust task and task difficulty based on patient progress (2f).

Well-being

1b.10 People with specific needs

VR offers a complementary approach to traditional therapy for individuals with special needs (i.e. children with the autism spectrum disorder) by providing simulated scenarios that help them better handle social (and working) situations strengthening the inclusion in the day life activities. VR environments can be easily implemented and replicated, allowing for repeated practice and personalised settings, which is often challenging in real-life conditions. The use of VR scenarios as serious games can also enhance engagement and therapy adherence where needed.

Challenges and opportunities: Traditional procedures and support activities (in same case therapies) for individuals with special needs, like the one based on cognitive behavioural strategies, social skills programmes or motor training, often involve exposing individuals to real-life scenarios to teach autonomy. While effective, this can be challenging to implement and replicate consistently in real life. Current VR solutions for this purpose are primarily initial proofs of concept and are not yet widely adopted as a standard of care.

Research and Innovation Objectives: Develop commercial head-mounted displays (and widely interaction devices) suitable for therapeutic applications and addressing the special needs of individuals (2a). Create VR and mixed reality (MR) customised applications specifically designed for strengthening social, emotional and cognitive skills (2b, 2c). Establish robust back-end platforms to support these applications (2d, 3b).

1b.11 Virtual communities for individuals recovering from temporary illness

XR technologies can create shared virtual spaces where patients recovering at home from temporary illnesses can connect with others, participate in social and meaningful activities, and mitigate feelings of isolation, boredom, or emotional distress. XR fosters a sense of presence and normality, thereby enhancing recovery motivation and psychological well-being.

Challenges and opportunities: Individuals recovering at home from temporary illnesses often rely on limited social support from family or friends, leading to feelings of disconnection and reduced mood or motivation. Current passive entertainment options offer minimal interaction or stimulation. While some rehabilitation programmes use basic applications or video calls, immersive community-based XR platforms for short-term recovery are not yet widely adopted.

Research and Innovation Objectives: Develop lightweight, easy-to-use XR systems suitable for home environments (2a, 2b). Design intuitive onboarding processes for users experiencing temporary fatigue or pain (2c). Incorporate asynchronous participation options to accommodate varying patient schedules and energy levels (2f, 2d).

1b.12 Education for clinical personnel

Technology-based education aims to raise awareness among clinical personnel regarding the current possibilities offered by the market and research centres in XR and VW. This enables informed decision making regarding technology, solutions, and their application in patient care, ensuring that healthcare professionals are equipped to integrate innovative approaches.

Challenges and opportunities: While initial courses on innovative technologies are emerging, they are often elective and not widely integrated into standard curricula across European Union countries. A dedicated education pathway for XR and VW technologies is currently lacking in standard medical training, and there is an absence of specific "hybrid" professional roles within hospitals that combine clinical expertise with technological understanding.

Research and Innovation Objectives: Develop educational programmes that inform clinical personnel about the opportunities, barriers, and ethical implications of new technologies. Establish mechanisms for continuous professional development to ensure healthcare providers remain updated on evolving technological advancements (1e, 3b, 3d).

Recommendations

To fully realise the transformative potential of VW in healthcare and well-being, a multi-faceted approach addressing technological, educational, and ethical considerations is essential. Firstly, there is a pressing need for the development of highly accessible and affordable VW hardware and software solutions. This includes lightweight, user-friendly XR devices suitable for diverse user groups, from elderly patients to individuals with specific needs. The widespread adoption of these technologies' hinges on their ease of use and economic viability, particularly for home-based applications and community-level interventions.

Secondly, significant investment is required in the creation of high-fidelity, evidence-based VW applications for prevention, diagnosis, personalised care, surgery, and rehabilitation. These applications must be rigorously validated against established clinical standards to ensure their efficacy and safety. A focus on gamification and engaging user experiences will be crucial for maintaining user motivation and adherence, particularly in long-term preventive and rehabilitative programmes.

Thirdly, the integration of AI and DT technologies is paramount for advancing personalised healthcare. Developing patient-specific DTs capable of real-time monitoring, predictive modelling, and tailored treatment recommendations will revolutionise clinical decision-making. This necessitates robust data integration from various sources, including wearable sensors and ingestible devices, alongside the development of intuitive XR interfaces for healthcare professionals.

Fourthly, addressing the educational gap among clinical personnel regarding XR and VW technologies is critical. Comprehensive educational programmes should be developed and integrated into standard medical curricula to ensure healthcare professionals are equipped to understand, evaluate, and apply these innovative approaches in patient care. The establishment of "hybrid" professional roles combining clinical expertise with technological understanding could further accelerate adoption.

Finally, ethical considerations, particularly concerning data privacy, consent, and the responsible use of AI in healthcare, must be embedded into the design and deployment of all VW solutions. Transparent data governance frameworks and robust safety-critical systems are essential to build trust and ensure that these technologies serve the public interest without exacerbating inequalities or causing harm.