1f. Security and Defence

Scope and Introduction

This chapter explores the critical research topics within the domain of security and defence in Virtual Worlds (VW), highlighting their strategic importance, defining existing challenges and research gaps, and outlining key objectives for future innovation to safeguard these emerging digital frontiers.

The application scenarios for VW in security and defence are diverse, ranging from training and simulation environments to advanced mission planning, on-field operational assistance, situational awareness for operational command, remote maintenance and technical support, and medical support and recovery. Each of these areas leverages the unique capabilities of VW to enhance preparedness, improve decision-making, and increase the safety and efficiency of personnel in high-stakes environments.

However, the effective deployment of VW technologies in security and defence is contingent upon addressing several key challenges. These include ensuring technological autonomy to mitigate reliance on non-European providers, developing robust hardware designed for demanding operational conditions, the ability to create cost-effective and customisable immersive content and the responsible integration of AI. Finally, while offering immense potential, the convergence of advanced digital technologies and decentralised systems also creates new attack surfaces and necessitates novel approaches to cybersecurity.

Use Cases

This section outlines key research areas within the security and defence domain, detailing their significance, current challenges, and proposed objectives for future innovation.

1f.1 Training and simulation

VW offer unparalleled opportunities for realistic and cost-effective military and civil protection training and simulation. This enables personnel to practice complex operations, including intervention, combat, piloting vehicles, behavioural skills, command, equipment maintenance, and medical interventions. Training can occur in shared, persistent virtual environments that replicate operational theatres, allowing users to connect via XR headsets, haptics, and voice systems from multiple locations. This leads to improved learning curves, enhanced on-field efficiency, greater diversity of training scenarios, and reduced training time and costs.

Challenges and opportunities: Traditional training systems often lack adaptability and immersion, with cost limiting the frequency and scope of exercises. While some simulators, especially for aviation, leverage VR or MR devices, many still rely on legacy means, such as screens or actual equipment. Overall adoption of VW solutions is modest, with full integration remaining at the research and development stage. Challenges include the expense and immaturity of XR hardware and ensuring user acceptance.

Research and Innovation Objectives: Create cost-effective, low-code/no-code content authoring tools that allow subject-matter experts to create training content by themselves (2c). Develop Live-Virtual-Constructive interoperability frameworks. Create cost-effective and user-friendly XR devices with extended battery life (2a). Develop AI-controlled virtual agents and adversaries (2f).

1f.2 Mission planning

VW provide immersive environments for team-based mission planning, typically supported by mixed reality setups. This allows commanders, field operators, and specialised units to test and plan missions before engagement, leveraging Digital Twins (DT) and simulations. The enhanced immersion and access to comprehensive data within these environments help to reduce cognitive overload and improve understanding of mission challenges.

Challenges and opportunities: Current mission planning relies on a combination of paper and digital tools, such as integrated Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance systems. While these tools utilise DTs and simulations, they often lack the immersive qualities needed to fully convey the complexities and challenges of a mission, potentially leading to cognitive overload and difficulties in understanding the operational environment.

Research and Innovation Objectives: Develop high-fidelity 3D environments for detailed mission visualisation (2a). Create cost-effective and user-friendly XR devices for planning (2a). Establish secured interoperability XR frameworks to facilitate collaboration (2d).

1f.3 On-field operational assistance

XR technologies can overlay mission-critical information directly into a user's field of view, providing contextual real-time guidance. This allows field personnel, such as soldiers or firefighters, to receive data on maps, threat indicators, biometrics, environmental alerts, friendly unit positions, and even facial or object recognition, while maintaining hands-free situational awareness. Coupled with AI-based assistants, these devices can filter and prioritise information and generate local analysis, significantly improving situational awareness, decision-making, coordination, and overall operational safety.

Challenges and opportunities: Current operational support relies on radios, handheld devices, or tablets, often leading to cognitive overload, distraction, degraded reactivity, poor coordination, and fragmented data visualisation. There is a growing need for real-time contextual data delivered to personnel on the ground. Adoption in operational contexts remains low, with pilot programmes facing setbacks due to hardware limitations.

Research and Innovation Objectives: Develop lightweight, ruggedised AR headsets with extended battery life suitable for field conditions (2a). Create AI-based context-aware assistants for real-time decision support (2f). Establish secure, high-bandwidth communication networks (4b).

1f.4 Situational awareness for operational command

XR enables commanders to visualise battlefield scenarios in real-time using 3D operational maps and immersive simulations, improving coordination between tactical and strategic levels. Virtual presence technologies allow for seamless collaboration between in-person and remote participants. Ultimately, this aims to create virtual spaces for advanced command and control centres accessible from anywhere, providing all stakeholders with relevant data and reducing the need for centralised physical locations.

Challenges and opportunities: Traditional command centres rely on in-person locations with multiple monitors and personnel, which can limit collaboration. While virtual presence and XR technologies are gaining traction in Europe, widespread deployment requires improved headsets, along with better integration with legacy systems. Holographic display adoption is also in early stages.

Research and Innovation Objectives: Develop high-fidelity 3D environments for comprehensive situational awareness (2a). Create cost-effective and user-friendly XR devices (2a). Establish secured interoperability XR frameworks to facilitate collaboration (2d).

1f.5 Remote maintenance and technical support

XR enables immersive and contextual support for maintenance tasks. Through AR/MR goggles, technicians can receive over-the-shoulder assistance, view 3D holographic models overlaid on physical components, and even access local, step-by-step procedures without connectivity. AI assistants, connected to camera streams, can aid in troubleshooting and prevent manipulation errors. This improves speed, accuracy, and safety, while reducing downtime and the need to deploy experts in the field.

Challenges and opportunities: Current maintenance relies on paper manuals and fragmented expert support via phone or videoconferencing, leading to slow troubleshooting, high error rates, and lengthy downtime for critical systems. Challenges include the need for improved XR goggles with high precision spatial anchoring, and the high cost of content creation for mixed reality-guided procedures. While AR-based remote assistance is used in civil aerospace and adapted for military use, widespread adoption in European defence forces remains low to moderate.

Research and Innovation Objectives: Develop high-fidelity AR/VR/XR goggles with low latency and high precision spatial anchoring, ensuring device security and ruggedisation (2a). Create cost-effective, low-code/no-code content authoring tools that allow subject-matter experts to reuse training content (2c). Establish secure and low-bandwidth remote collaboration protocols and ensure numerical continuity through integration with asset management systems and DTs (2d).

1f.6 Medical support and recovery

VW enable comprehensive medical and psychological intervention training, including combat trauma care, field surgery, post-traumatic stress disorder therapy and operational decompression. Shared immersive scenarios, dynamic emotional stress simulations, and scenario branching based on user biometrics enhance realism, leading to improved memory consolidation and procedural fluency. Empathetic and emotionally adaptive therapy scenarios enhance post-traumatic stress disorder treatment outcomes, while real-time adjustment based on stress and muscle activation signals further refines interventions.

Challenges and opportunities: Traditional medical training relies on role-playing and screen-based trainers, which offer limited realism and adaptability to stress or skill levels. Physical simulations are costly and lack emotional realism. While virtual reality-based post-traumatic stress disorder therapy and trauma simulation are gaining traction, full virtual world-based Live-Virtual-Constructive integration for medical support remains at a low adoption level.

Research and Innovation Objectives: Develop real-time biofeedback systems (e.g., heart rate variability, electroencephalography, galvanic skin response) and stress modelling for personalised trauma scenarios (2a). Develop low-code authoring tools and representative 3D environments (2c). Develop AI-driven virtual avatars for guidance and support, along with adaptive multimodal interfaces (voice, gesture, haptics) (2f).

1f.7 Human-Machine Interaction (HMI)

Using XR technologies, operators can "become" the drone or robot, controlling autonomous systems through immersive 3D interfaces and natural gestures or voice commands. Situational data, drone feeds, or robot status can be overlaid directly onto the real environment, improving contextual awareness and decision-making. Haptic controllers can simulate forces and resistance, such as wind, recoil, collisions, or uneven terrains, enhancing the realism and precision of remote operations.

Challenges and opportunities: Current unmanned systems are often controlled via rugged tablets, joysticks, or dedicated stations, which require significant training and limit situational awareness. First-person goggles used for drone control typically provide only an immersive video feed with limited spatial awareness, lacking depth perception and haptic feedback. These interfaces contribute to high cognitive load and difficulty in navigating complex terrains. While some advanced first-person view systems are evolving towards XR by integrating overlays or gesture-based control, overall operational use of XR interfaces remains at a low adoption level.

Research and Innovation Objectives: Develop intuitive multimodal human-machine interfaces (2b). Create real-time data fusion and visualisation capabilities (2a). Implement AI-assisted decision support and target recognition (2f).

Key Challenges

1f.8 Ensuring technological autonomy

Europe faces a critical strategic dependency on non-European providers for core XR technologies, including processors, sensors, and software for authoring and rendering. Pursuing technological autonomy in the XR domain is crucial for safeguarding data security and integrity, mitigating supply chain vulnerabilities, and ensuring that European forces are not locked into non-sovereign ecosystems. This requires strategic efforts across the full XR value chain to develop European-designed components, open software platforms, and secure frameworks.

Challenges and opportunities: The XR market is dominated by non-European hardware and software providers, leading to concerns regarding cybersecurity, technological and data autonomy, and long-term autonomy in critical sectors for the European Union. This reliance poses risks of surveillance or manipulation of sensitive defence data, potential restrictions on access to key XR components during geopolitical tensions, and the risk of being locked into non-European standards.

Research and Innovation Objectives: Develop European-designed processors, sensors, and headsets. Create open, certifiable, and high-performance software platforms (rendering engines, authoring tools, simulation frameworks). Establish European XR standards and certification frameworks.

1f.9 Hardware for Extended Reality systems

The deployment of VW technologies in security and defence settings necessitates the development of hardware that goes beyond consumer-grade solutions. This includes creating cost-effective, user-friendly, and lightweight XR devices with low-latency displays and high-precision spatial anchoring. Specifically for this application domain, ruggedised and secured XR headsets are crucial for achieving operational readiness and scale in demanding environments.

Challenges and opportunities: Current XR hardware often lacks the specific characteristics required for security and defence applications, such as ruggedisation, enhanced security, and extended battery life.

Research and Innovation Objectives: Develop cost-effective and user-friendly, lightweight XR devices with low latency displays and high precision spatial anchoring. Create ruggedised and secured XR headsets suitable for harsh operational environments (2a).

1f.10 Immersive content authoring

Enabling cost-effective content creation accessible to subject-matter experts, through low-code/no-code editors, is crucial for the widespread adoption of VW technologies in security and defence. This allows for the rapid development and customisation of mission-relevant interactive environments and the reuse of content across different use cases. Such tools empower operational personnel to create tailored training scenarios and mission plans without extensive programming knowledge.

Challenges and opportunities: The creation of high-fidelity immersive content for security and defence applications is currently expensive and requires specialised technical expertise. There is a lack of user-friendly authoring tools that allow subject-matter experts to directly contribute to content development, limiting the agility and relevance of training and simulation environments.

Research and Innovation Objectives: Develop cost-effective low-code/no-code content authoring tools that are accessible to subject-matter experts. Design tools facilitating the reuse of immersive content across various use cases, such as training and field support (2c).

1f.11 User interaction and ergonomics

Effective deployment of XR systems in security and defence requires advanced user interfaces and user experiences that enable rapid cognition without distraction. Adaptive multimodal interfaces, incorporating voice, gesture, and haptics, are essential for intuitive and efficient interaction in high-stress environments. Ensuring ergonomic comfort for prolonged use in harsh and variable field conditions is also critical to prevent fatigue or injury and enhance human performance.

Challenges and opportunities: Current XR interfaces may not be optimised for the rapid decision-making and high-pressure scenarios typical of security and defence operations, potentially leading to cognitive overload.

Research and Innovation Objectives: Develop advanced user interface/UX designs that facilitate rapid cognition in XR environments. Create adaptive multimodal interfaces that seamlessly integrate voice, gesture, and haptics for intuitive control. Conduct human factors research to ensure ergonomic comfort for prolonged use (2b).

1f.12 Artificial Intelligence (AI) and agents

The integration of AI and intelligent agents is vital for enhancing the realism and effectiveness of VW in security and defence. AI-based context-aware assistants can provide real-time decision support, while AI-controlled virtual agents and adversaries enable highly dynamic and realistic training scenarios. AI-assisted content creation streamlines the development of complex environments, and AI-monitored performance with adaptive tutoring can tailor scenarios on the fly, improving learning outcomes.

Challenges and opportunities: The development of highly intelligent and adaptable AI agents for realistic simulation in security and defence contexts remains a significant challenge. Ensuring that AI systems provide reliable and trustworthy decision support in high-stakes situations requires robust validation and ethical considerations.

Research and Innovation Objectives: Develop AI-based context-aware assistants for real-time decision support. Create AI-controlled virtual agents and adversaries that exhibit realistic and adaptive behaviours for training and simulation. Develop AI-assisted tools to streamline the content creation process. Implement AI-monitored performance assessment and adaptive tutoring systems to personalise training scenarios (2f).

1f.13 Systems integration, networks, and infrastructure

Secure and robust communication networks are foundational for the effective deployment of VW technologies in security and defence, enabling seamless data exchange and real-time collaboration across distributed teams. Establishing a distributed and secured collaborative XR framework, including Live-Virtual-Constructive interoperability, is crucial for integrating diverse operational elements into a unified environment.

Challenges and opportunities: The integration of VW technologies into existing security and defence infrastructures presents significant challenges, particularly concerning network bandwidth, latency, and cybersecurity. Ensuring secure and robust communication in contested environments is paramount.

Research and Innovation Objectives: Develop secure and robust communication networks capable of supporting high-bandwidth, low-latency XR applications in challenging environments. Create distributed and secured collaborative XR frameworks, including Live-Virtual-Constructive interoperability protocols. Ensure numerical continuity through seamless integration with asset management systems and DTs (2d).

1f.14 Human monitoring

Integrating human monitoring capabilities, such as biometrics and cognitive load monitoring, into VW is essential for enhancing human performance and ensuring the well-being of operators in high-stress security and defence environments. Multimodal biometric fusion for scenario adaptation allows training environments to dynamically respond to an individual's physiological and cognitive state, optimising learning and performance.

Challenges and opportunities: The effective deployment of XR technologies in security and defence depends fundamentally on their acceptance and impact on human operators. Systems must be designed to enhance human performance without causing cognitive overload or unintended psychological harm. Ethical considerations, particularly regarding the use of AI-driven virtual agents, biometric monitoring, and behavioural data, raise concerns about surveillance, autonomy, and consent.

Research and Innovation Objectives: Develop advanced biometrics and cognitive load monitoring systems for real-time assessment of operator state in XR environments. Create multimodal biometric fusion techniques for dynamic scenario adaptation in training and operational support. Research and implement emotion recognition capabilities via facial, gaze, and vocal analysis for enhanced human-machine interaction (2a).

1f.15 Economic and market challenges

Europe's fragmented defence market, characterised by diverse national requirements and procurement rules, creates significant economic barriers to the adoption of XR technologies, in contrast with countries with a very large domestic market such as China and US. Fostering greater cooperation between member states, harmonising procurement processes, and promoting coordinated investments can create larger, integrated markets.

Challenges and opportunities: The fragmentation of the European defence market complicates joint procurement and increases costs. This makes it more difficult for the European industry to foster global leaders.

Research and Innovation Objectives: Foster greater cooperation among member states to harmonise procurement processes and promote coordinated investments to create larger, integrated markets. Development of common standards for interoperability and duplication reduction will further contribute here (2d).

Recommendations

To ensure the effective, secure, and sovereign deployment of VW technologies in security and defence, a comprehensive set of recommendations is proposed, addressing key challenges across technology, economics, and human factors.

Technological Autonomy: A strategic imperative is to achieve technological autonomy in XR. This necessitates a concerted European effort to develop and manufacture core XR components, including processors, sensors, and headsets. Furthermore, the creation of open, certifiable, and high-performance software platforms, such as rendering engines, authoring tools and simulation frameworks, is crucial to reduce reliance on non-European providers.

Purpose-Built Hardware: Significant investment is required in the development of purpose-built hardware for XR systems. This includes creating cost-effective, user-friendly, lightweight, and ruggedised XR devices with low-latency displays and high-precision spatial anchoring, suitable for harsh operational environments.

Content Authoring: Enabling cost-effective and accessible immersive content authoring is crucial. This involves developing low-code/no-code editors that empower subject-matter experts to create and customise interactive, realistic, mission-relevant environments. Designing frameworks that facilitate the reuse of immersive content across various use cases will significantly reduce development time and costs.

User Experience: A strong focus on user interaction and ergonomics is essential. Advanced user interface/UX designs that facilitate rapid cognition and minimise distraction in high-stress XR environments must be developed. Creating adaptive multimodal interfaces incorporating voice, gesture, and haptics will ensure intuitive and efficient control.

AI Integration: The responsible integration of AI and intelligent agents is paramount. This includes developing AI-based context-aware assistants for real-time decision support, creating AI-controlled virtual agents and adversaries for realistic training scenarios, developing AI-assisted tools to streamline the content creation process, and implementing AI-monitored performance assessment and adaptive tutoring systems.

Infrastructure: Ensuring secure and robust systems integration, networks, and infrastructure is foundational. This requires developing secure and high-bandwidth communication networks capable of supporting low-latency XR applications in challenging environments. Establishing distributed and secured collaborative frameworks, including Live-Virtual-Constructive interoperability protocols, will create truly integrated operational environments.

Human Factors: Integrating human monitoring capabilities, such as biometrics and cognitive load monitoring, is essential for enhancing human performance and ensuring the well-being of operators in high-stress environments, necessitating transparent data governance and ethical frameworks.

Market Integration: Addressing economic and market fragmentation is vital for scaling the adoption of XR in defence. Greater cooperation among member states to harmonise procurement processes and promote coordinated investments will create larger, integrated markets. The development of common standards and certification frameworks will facilitate interoperability, reduce duplication, and enhance the global competitiveness of European industries.