1b. Healthcare and Well-being

Scope and Introduction

Virtual Worlds (VW) technologies are transforming healthcare delivery through personalised prevention, early detection, tailored treatments, and enhanced rehabilitation methods. By integrating Extended Reality (XR), Digital Twins (DT), Artificial Intelligence (AI), and advanced sensing technologies, healthcare systems can provide more accessible, effective, and patient-centred care.

The healthcare sector faces critical challenges including aging populations, rising chronic disease burdens, limited access to specialists, and increasing costs. VW technologies offer innovative solutions by enabling remote monitoring, virtual diagnostics, personalised treatment planning, and engaging rehabilitation programmes that can be delivered at scale while maintaining quality of care.

Use Cases

This section outlines key research areas within the healthcare and well-being domain, detailing their significance, current challenges, and proposed objectives for future innovation.

Prevention

1b.1 Virtual Worlds for promoting healthy lifestyles

Virtual environments can encourage individuals to adopt healthier habits through engaging, gamified experiences that track and respond to daily activities. By providing real-time feedback on fitness progress, nutrition, and sleep patterns within immersive settings, VW can motivate sustained behaviour change and social support through shared virtual fitness communities.

Challenges and opportunities: Current fitness apps and wearable devices provide limited interactivity and social engagement. Individuals often struggle to maintain motivation for long-term health behaviour change. Existing digital health solutions lack the immersive, emotionally engaging experiences needed to sustain user interest. The integration of gamification with personalised health data remains underdeveloped, and social support mechanisms are fragmented across platforms.

Research and Innovation Objectives: Develop immersive VW applications that integrate real-time health data from wearable devices (2a, 2e). Create AI-driven personalisation systems that adapt fitness programmes to individual needs and progress (2f). Design social VW environments that foster community support and shared fitness goals (2b, 2c). Implement robust privacy frameworks for health data management (3b). Establish evidence-based gamification strategies that maintain long-term user engagement (2c).

1b.2 Extended Reality for fostering social connections

XR technologies enable elderly individuals and those with limited mobility to participate in virtual social activities, reducing isolation and improving mental well-being. By creating accessible virtual gathering spaces for family interactions, community events, and shared hobbies, XR can enhance quality of life and maintain social bonds across geographic distances.

Challenges and opportunities: Social isolation among elderly populations is a significant public health concern, particularly post-pandemic. Current digital communication tools lack the sense of presence and emotional engagement of physical interaction. Accessibility barriers prevent many elderly users from adopting existing VR/AR technologies. There is limited integration between healthcare monitoring and social VW platforms, missing opportunities for holistic wellbeing support.

Research and Innovation Objectives: Develop age-appropriate, intuitive XR interfaces with minimal technical barriers (2a, 2b, 2c). Create accessible hardware solutions suitable for elderly users and those with physical limitations (2a). Design social VW spaces that support meaningful interaction and emotional connection (2b). Integrate health monitoring with social engagement platforms to provide holistic wellbeing support (2e, 2f). Establish privacy protections and consent mechanisms appropriate for vulnerable populations (3b, 3a).

1b.3 Virtual Worlds for cardiovascular health monitoring

A comprehensive VW environment can integrate multiple wearable devices and ingestible sensors (smart pills) to continuously monitor cardiovascular parameters, creating a holistic view of an individual's heart health. This enables early detection of abnormalities, lifestyle adjustments to prevent disease progression, and provides continuous information flow for medical professionals.

Challenges and opportunities: Individuals often lack awareness of cardiovascular health risks in the absence of symptoms. Current wearable device data is often unintelligible and incomprehensible to users. There is insufficient integration between consumer health devices and clinical monitoring systems. Early detection systems for cardiovascular disease require more sophisticated data analysis and predictive modelling. Creating awareness and motivating lifestyle changes remains a significant challenge.

Research and Innovation Objectives: Develop monitoring systems that integrate DT with wearable sensors for continuous surveillance of user parameters (2e, 2d). Advance data analysis and risk prediction modelling (2f). Implement augmented warning systems for changes in physiological parameters linked to cardiac disease or heart failure (2a, 2b). Create intuitive visualisations that help individuals understand their cardiovascular health status (2c). Establish secure, privacy-preserving data sharing protocols between patients and healthcare providers (3b).

Personalised Care

1b.4 Personalised treatment through patient Digital Twins

Developing accurate patient-specific DT using advanced system identification and modelling techniques assists clinicians in predicting disease progression, personalising treatments, and optimising healthcare outcomes. This allows for virtual testing and refinement of different treatments, leading to optimised therapeutic strategies, reduced uncertainty, minimised side effects, and improved patient engagement in care decisions.

Challenges and opportunities: Current treatment planning relies heavily on general clinical guidelines, population-level statistics, and limited patient-specific data, leading to clinical uncertainty and trial-and-error methods. Existing patient-specific modelling methods often lack accuracy, adaptability, and scalability for clinical practice. There are gaps in utilising advanced identification methods, augmented sensing, and integrating physics-inspired data blocks crucial for patient-specific DT development.

Research and Innovation Objectives: Develop patient-specific DT integrated into comprehensive personal care VW connected to networks of wearable and environmental sensors (2a, 2d, 2e, 2f). Implement secure data management systems that provide privacy while maintaining information flow across the medical care chain (2d, 3b). Create specialised interfaces for clinical staff and customised interfaces for explaining care plans to patients (2b). Advance predictive modelling capabilities for disease progression and treatment outcomes (2f).

1b.5 Adaptation of Virtual Worlds according to patient needs (AI and DT-Supported)

Digital twins as virtual replicas of patients are transforming personalised healthcare by enabling real-time monitoring, predictive modelling, and tailored treatment plans. XR technologies enhance personalisation, emotional engagement, and accessibility for elderly individuals with cognitive impairment or children/adults with special needs. This approach can reduce reliance on sedatives, improve therapeutic outcomes, and promote emotional wellbeing and cognitive engagement.

Challenges and opportunities: Existing patient DT examples often focus on specific topics and lack interoperability for clinical specialists from diverse backgrounds. Integration of DTs into routine clinical practice is still in early stages. Elderly individuals often receive sedatives to manage discomfort or participate in traditional group-based reminiscence therapy, which is less personalised and engaging. Conventional methods face limitations including reduced accessibility, limited individualisation, dependence on facilitator skills, low sensory stimulation, and difficulties for participants with cognitive or communication impairments.

Research and Innovation Objectives: Develop patient-specific DT integrated into comprehensive personal care VW connected to sensor networks in wearable and living environments (2a, 2d, 2e, 2f). Implement secure, privacy-preserving data management systems (2d, 3b). Design specialised interfaces for different user groups including individuals, families, and clinical staff (2b). Create adaptive XR environments that respond to cognitive abilities and emotional states (2f). Establish evidence-based protocols for XR-enhanced therapy for diverse patient populations (3a, 3e).

Surgery

1b.6 Real-time intraoperative Digital Twin for surgical assistance

This system receives real-time sensor data during surgery and continuously predicts tissue response, suggesting optimal surgical paths. It acts as a closed-loop assistant, providing warnings and suggesting adjustments, and can be visualised in XR to give surgeons intuitive feedback, ultimately enhancing surgical safety and precision.

Challenges and opportunities: Current surgical workflows primarily rely on static pre-operative imaging and intraoperative visual interpretation. While some robotic and image-guided tools offer updated navigation, they lack dynamic, patient-specific physiological models that adapt to surgical events. Real-time feedback or predictive control based on continuously updating DTs remains at an early research stage. Intraoperative decisions still heavily rely on surgeon experience. Gaps exist in using real-time identification methods, developing hybrid models, fusing sensory data, and creating predictive control loops.

Research and Innovation Objectives: Develop hybrid and data-driven modelling techniques for real-time surgical support (2d, 3b). Advance system identification and dynamic modelling of soft tissues (2e, 2f). Implement real-time control loops, model updates, and feedback mechanisms with sensor fusion capabilities (2a, 2e, 2f). Create XR visualisation with low-latency interaction for surgical environments (2b, 2c). Establish safety-critical systems with fail-safe modes for surgical applications (2a, 3b). Ensure robust validation and regulatory compliance for intraoperative AI systems (3e).

1b.7 Endoscopic inspection as a diagnostic procedure

Enhancing information available to physicians through advanced image sensors and multi-modality imaging systems, potentially combined with smart pill cameras, paves the way for patient-specific treatment planning. Integration with XR can make procedures faster, more accurate, and with fewer side effects by improving surgical and endoscopic outcomes, providing intraoperative guidance, and monitoring smart pill passage.

Challenges and opportunities: Challenges include the miniaturisation of imaging sensors, integration of multi-modal data facilitated by AI, and provision of AI-based decision support for diagnostics and treatment planning. Embedding real-time information during diagnostic and therapeutic interventions and combining it with previously collected data (e.g., live endoscopic imaging with smart pill images) remain key hurdles. While endoscopic procedures are widely adopted, these enhancements represent significant improvements within existing practices.

Research and Innovation Objectives: Develop miniaturised imaging sensors, including hyperspectral imaging, to provide high-quality imaging and video at relevant wavelengths (2a). Integrate multi-modal sensor data fusion for enhanced decision support (2a, 2f). Create XR-enhanced, intuitive visualisations and interfaces for surgeons and physicians (2b, 2c). Develop AI models and DTs for training, learning, and surgery planning (2f, 2e). Establish protocols for real-time data integration and clinical decision support (2d).

Rehabilitation – Therapy Monitoring

1b.8 Extended Reality technology to support rehabilitation intervention

Personalised and adaptive XR environments can support rehabilitative interventions, even without constant therapist supervision. This allows for longer training periods, potentially in a combined modality where XR acts as an add-on to standard therapy, enabling more patients to receive intervention simultaneously, even at home. Multi-user environments could further enhance adherence and motivation.

Challenges and opportunities: Therapists typically provide one-to-one sessions, often limited by strict patient schedules and time constraints, requiring the therapist to be present throughout. While some XR solutions are applied in research hospitals, they are not widely adopted, and patients often receive a limited number of scheduled sessions before discharge. Home-based rehabilitation lacks the monitoring and adaptation that clinical settings provide.

Research and Innovation Objectives: Develop specialised rehabilitation applications with evidence-based therapeutic protocols (2b, 2c). Create lightweight, wearable hardware devices suitable for home use (2a). Design user-friendly interfaces for patients and remote monitoring interfaces for therapists (2c). Implement adaptive mechanisms to adjust task difficulty based on patient progress using AI (2f). Establish protocols for remote supervision and intervention when needed (2d, 3b). Validate clinical efficacy through rigorous trials (3e).

Well-being

1b.9 People with specific needs

VR offers a complementary approach to traditional therapy for individuals with special needs (e.g., children with autism spectrum disorder) by providing simulated scenarios that help them better handle social and working situations, strengthening inclusion in daily activities. VR environments can be easily implemented and replicated, allowing for repeated practice and personalised settings, which is often challenging in real-life conditions. The use of VR scenarios as serious games can enhance engagement and therapy adherence.

Challenges and opportunities: Traditional procedures and support activities for individuals with special needs, based on cognitive behavioural strategies, social skills programmes, or motor training, often involve exposing individuals to real-life scenarios to teach autonomy. While effective, this can be challenging to implement and replicate consistently. Current VR solutions for this purpose are primarily initial proofs of concept and not yet widely adopted as standard care.

Research and Innovation Objectives: Develop commercial head-mounted displays and interaction devices suitable for therapeutic applications addressing the special needs of individuals (2a). Create VR and mixed reality customised applications specifically designed for strengthening social, emotional, and cognitive skills (2b, 2c). Establish robust back-end platforms to support these applications (2d, 3b). Conduct rigorous clinical validation studies (3e). Ensure accessibility and safety for diverse user populations (3a).

1b.10 Virtual communities for individuals recovering from temporary illness

XR technologies can create shared virtual spaces where patients recovering at home from temporary illnesses can connect with others, participate in social and meaningful activities, and mitigate feelings of isolation, boredom, or emotional distress. XR fosters a sense of presence and normality, thereby enhancing recovery motivation and psychological well-being.

Challenges and opportunities: Individuals recovering at home from temporary illnesses often rely on limited social support from family or friends, leading to feelings of disconnection and reduced mood or motivation. Current passive entertainment options offer minimal interaction or stimulation. While some rehabilitation programmes use basic applications or video calls, immersive community-based XR platforms for short-term recovery are not yet widely adopted.

Research and Innovation Objectives: Develop lightweight, easy-to-use XR systems suitable for home environments (2a, 2b). Design intuitive onboarding processes for users experiencing temporary fatigue or pain (2c). Incorporate asynchronous participation options to accommodate varying patient schedules and energy levels (2f, 2d). Create supportive community moderation and mental health monitoring systems (3a, 3b). Establish protocols for escalation to professional support when needed (3e).

1b.11 Education for clinical personnel

Technology-based education aims to raise awareness among clinical personnel regarding current possibilities offered by the market and research centres in XR and VW. This enables informed decision-making regarding technology, solutions, and their application in patient care, ensuring healthcare professionals are equipped to integrate innovative approaches.

Challenges and opportunities: While initial courses on innovative technologies are emerging, they are often elective and not widely integrated into standard curricula across European Union countries. A dedicated education pathway for XR and VW technologies is currently lacking in standard medical training, and there is an absence of specific "hybrid" professional roles within hospitals that combine clinical expertise with technological understanding.

Research and Innovation Objectives: Develop educational programmes that inform clinical personnel about opportunities, barriers, and ethical implications of new technologies (1e). Establish mechanisms for continuous professional development to ensure healthcare providers remain updated on evolving technological advancements (3b, 3d). Create training pathways for hybrid clinical-technical roles (1e). Develop evidence-based frameworks for technology assessment and adoption in clinical settings (3e).

Recommendations

To fully realise the transformative potential of VW in healthcare and well-being, a multi-faceted approach addressing technological, educational, and ethical considerations is essential.

Accessible and Affordable Solutions: There is a pressing need for developing highly accessible and affordable VW hardware and software solutions. This includes lightweight, user-friendly XR devices suitable for diverse user groups, from elderly patients to individuals with specific needs. Widespread adoption hinges on ease of use and economic viability, particularly for home-based applications and community-level interventions.

Evidence-Based Applications: Significant investment is required in creating high-fidelity, evidence-based VW applications for prevention, diagnosis, personalised care, surgery, and rehabilitation. These applications must be rigorously validated against established clinical standards to ensure efficacy and safety. A focus on gamification and engaging user experiences will be crucial for maintaining user motivation and adherence, particularly in long-term preventive and rehabilitative programmes.

AI and Digital Twin Integration: The integration of AI and DT technologies is paramount for advancing personalised healthcare. Developing patient-specific DTs capable of real-time monitoring, predictive modelling, and tailored treatment recommendations will revolutionise clinical decision-making. This necessitates robust data integration from various sources, including wearable sensors and ingestible devices, alongside the development of intuitive XR interfaces for healthcare professionals.

Clinical Education: Addressing the educational gap among clinical personnel regarding XR and VW technologies is critical. Comprehensive educational programmes should be developed and integrated into standard medical curricula to ensure healthcare professionals are equipped to understand, evaluate, and apply these innovative approaches in patient care. The establishment of "hybrid" professional roles combining clinical expertise with technological understanding could further accelerate adoption.

Ethical Framework: Ethical considerations, particularly concerning data privacy, consent, and the responsible use of AI in healthcare, must be embedded into the design and deployment of all VW solutions. Transparent data governance frameworks and robust safety-critical systems are essential to build trust and ensure that these technologies serve the public interest without exacerbating inequalities or causing harm.