2c. Authoring and Experience Design

Scope

Authoring and designing Virtual Worlds (VW) experiences are crucial for XR technology and market development. This involves creating and structuring interactive experiences that blend virtual and real worlds. Authors—including independent creators, artists, developers, studios, and companies—are vital; without them, there's no innovation, content, or sustainable infrastructure. Authoring acts as a strategic lever, intersecting technological challenges, creative uses, Web4 industrial structuring, and digital economy sustainability, forming a foundational element for all use-cases from Industry to Healthcare.

However, two major barriers impede progress: First, authoring VW experiences is highly complex and multidisciplinary, requiring mastery of multiple technologies, making it inaccessible to many. This hinders the democratisation of content creation, which is a significant driver of societal and economic development. Second, Europe's content production relies heavily on non-European authoring tools and content distribution infrastructures. This dependence imposes external standards and commercial constraints on Europe's industry and autonomy.

In the rapidly growing VW market, authoring and design are often overlooked, despite being critical for valuing digital experiences. While XR job growth is projected, authors are often marginalized in terms of value and resource access. The sector's complexity, fragmentation, and lack of standards hinder professionalisation, though European platforms show promise for collaborative design.

The dominance of non-European platforms creates unprecedented dependency, concentrating authoring tools and distribution. This limits creative diversity and traps communities in closed ecosystems, posing strategic, economic, and cultural risks.

Ultimately, experience design is key for widespread immersive technology adoption. The quality of interaction, relevance of narratives, and perceived usefulness depend on creative excellence and technological performance. Large-scale adoption demands engaging, accessible, and value-generating experiences, which require robust technologies and collaborative co-design platforms.

Key principle: Authoring must be recognised as a prerequisite for technology adoption and market development, not a secondary output, especially given Europe's opportunity in content-based services. Fostering innovative authoring conditions and tools is key to building a sovereign, sustainable European VW ecosystem.

---

Foundations for Authoring and Experience Design

Dependence on Closed Non-EU Ecosystems: EU Autonomy at Risk

Challenge: Immersive authoring largely depends on non-European tech providers who monopolize tools and distribution, imposing proprietary standards. This systemic dependency undermines Europe's ability to define its own standards, strategic autonomy, and to spread its cultural values. Lack of major European players hinders independent technical communities and viable sovereign ecosystems.

Industry and creators are forced to master foreign technologies, making Europe a follower susceptible to non-EU commercial restrictions (e.g., U.S. laws). Talent relocation is often prioritized, limiting transnational openness. Training in closed systems homogenizes practices, reducing diversity. Lack of interoperability impedes content circulation and innovation. This dependency also offshores talent and IP.

Strategic Actions:

• Support European leaders in XR authoring tools, content, and distribution to guarantee EU autonomy. Foster strong technical communities to achieve this.
• Develop a strategy to reach critical mass in resources for solutions and creators to drive adoption, ensuring ecosystem growth and independence.

Rebalance Market's Power Dynamics

Challenge: The immersive market sees power concentrated among a few non-European tech giants, limiting diversity and innovation. The sector is largely techno-centric, prioritizing features over impactful content. This impedes adoption and strains authors with fragmented, complex tools. Studies on authoring methodologies and authoring working conditions are lacking. This concentration fosters predatory practices and precarious conditions for creators.

Tech actors control authoring solutions and communities, making authors dependent on proprietary technologies. These communities operate in closed ecosystems where training and resources are controlled by dominant players. Lack of independent, technology-agnostic communities obstructs a healthy, dynamic market.

Strategic Actions:

• Conduct global studies on VW creators' labour conditions to inform policies promoting equitable, sustainable creative ecosystems
• Conduct strategic research to dynamically map platform ecosystems, their territoriality, and automate key actor monitoring
• Investigate, support, and design independent, technology-agnostic collectives and communities to foster open, diverse digital ecosystems and reduce proprietary platform dependency, promoting standardisation, openness, and interoperability

Empower Creation Capabilities

Challenge: The immersive experience sector's vitality relies on experimentation, prototyping, and rapid iteration based on user feedback in variable conditions. Three obstacles hinder this: First, insufficient support for experiential research limits exploration and prototyping, hindering innovative formats and risk-taking. Second, the lack of fast, shared analytical tools impedes iterative improvement based on user feedback. This limits understanding of user impacts and expectations, making it harder to guide creators toward successful productions. Third, content creation tools have yet to evolve to fully support the design of complex and collaborative experiences. Significant room remains for innovation to enhance their capabilities, expand creative practices, and enable a wider range of use cases and contributors.

Strategic Actions:

• Strengthen R&D capacities in immersive creation for non-commercial projects and increase direct funding for creators
• Support research programs on new forms of XR content at the intersection of academic, cultural, and industrial sectors, promoting iterative design through experimentation and prototyping
• Support research on tools that empower content creation tools
• Investigate the design and development of standardized assessment tools for early integration of user feedback

Structuring of Skills and Professions in a Multidisciplinary Sector

Challenge: A key challenge for VW design is its extreme multidisciplinary characteristic, driven by the immersive nature of VW. The shift from 2D to 3D/4D transforms design and UX profoundly. Spatial interface design requires diverse skills: 3D modelling, real-time programming (Unity, Unreal), immersive interaction design (spatial UX/UI), interactive storytelling, cognitive psychology, and sensor engineering. A multidisciplinary approach ensures coherent, high-performing, and engaging experiences.

Lack of structured learning and educational pathways is a major obstacle. Immersive technology training is fragmented, often limited to technical modules. The emerging immersive environment demands new professional roles. This gap contrasts with the sector's need for hybrid profiles. Recruiting high-level curriculum experts is costly, leading to missed opportunities and undermining training quality.

To offer interdisciplinary, project-based training aligned with economic needs, stronger bridges are needed with each sector (health, education, retail). This enables knowledge exchange between sector practices and creators. Mutual understanding and collaboration are essential for authors to gain real-world insights and for sectors to recognize immersive content's creative potential.

Strategic Actions:

• Investigate multidisciplinary curricula aligned with industry needs, in collaboration between academic institutions and industry
• Support capacity building and education to enable universities and training centres to attract top-tier immersive technology experts
• Foster scalable, accessible, and practice-oriented training models across Europe that respond to rapid evolution of immersive tools and workflows, addressing cost and accessibility barriers

---

Research Topics for Authoring and Distribution

2c.1 Development of European Authoring and Distribution Tools

Challenges and opportunities: Authoring and distribution platforms are strategically vital in the immersive technology value chain. Developing European solutions and preventing interdependence between authoring and distribution tools is crucial to address the current concentration of solutions in few hands and the dominance of non-European players. This approach is key to achieving industry standards, interoperability, and reclaiming strategic autonomy for Europe.

Problem Definition and Research Gap: The current ecosystem suffers from an unprecedented concentration of creation tools, standards, and distribution channels in the hands of a few dominant non-European players. Authoring platforms, though adapting to VW, are largely established 3D engines not specifically designed for XR, affecting content creators' competitiveness. Similarly, distribution platforms are often evolutions of broader platforms. This structural imbalance binds VW actors to technology providers' restrictive practices, limiting content circulation, diversity, and innovation. There is a strong need for European alternatives that foster open, interoperable solutions, unlike the current closed ecosystems.

Research and Innovation Objectives:

• Investigate tools, approaches, and methodologies to reduce structural interdependence between authoring and distribution, fostering a more modular and competitive immersive industry
• Support the unification of EU technological bricks towards open-source authoring and/or distribution solutions as collaborative ecosystems promoting transparency, modularity, and long-term accessibility
• The ambition is to develop European XR-dedicated authoring and distribution solutions, reduce dependence on non-European ecosystems, enhance market diversity, empower creators, and achieve industry standards and interoperability

2c.2 Tools and Metrics for Experience and Impact Evaluation

Challenges and opportunities: Establishing clear, unified metrics, KPIs, and KVIs for evaluating VW experiences is crucial. This helps quantify benefits, limitations, and ROI, addressing the current difficulty in predicting the concrete impacts of VW content. Providing tools to certify experience quality (utility, usability, ergonomics, efficiency, accessibility) is essential for accelerating VW adoption by making impacts predictable and measurable.

Problem Definition and Research Gap: Evaluating the impacts of VW is complex due to a lack of unified metrics, KPIs, and KVIs to quantify benefits, limitations, and ROI precisely. There are no tools to certify that a VW was experienced as designed, and no recognised frameworks to evaluate XR experience quality across various metrics like utility, usability, ergonomics, efficiency, and accessibility. The strong dependence on major technology providers means authoring solutions rely on their often-confidential data regarding experience and economic success. These gaps directly impede VW adoption as it's difficult to predict concrete, on-the-field impacts over short to long terms.

Research and Innovation Objectives:

• Explore subjective and objective metrics and techniques to quantify XR experience quality. Investigate how to make such tools accessible to VW creators to facilitate iterative design
• Perform short, mid, and long-term studies to validate metrics and better understand concrete impacts of deployed VW on end-user practices
• Explore how collaborative authoring and/or experiencing environments and integrated platforms can support early-stage evaluation, enabling user feedback and impact assessment directly within design workflows

---

Research Topics for Authoring Tools and Workflows

2c.3 Versioning and Development

Challenges and opportunities: Improving the collaborative design process for VW experiences by effectively managing multiple past and concurrent versions. This addresses the limitations of current 3D frameworks that offer poor support for collaborative design, hindering multidisciplinary teams. Facilitating robust versioning, replaying modifications, and re-using components would significantly enhance maintainability, evolution, and efficiency of VW experiences.

Problem Definition and Research Gap: A significant challenge for collaborative design of VW experiences is managing multiple past and concurrent versions. Current major 3D frameworks like Unity offer poor or no direct support for collaborative 3D scene design. Challenges include tracking, documenting, debugging and replaying modifications, easily switching design alternatives, and merging versions with precise control. Git-based versioning is adopted but offers limited support for the multidisciplinary, non-developer profiles needed for VW authoring. Facilitating the re-use of previously designed components remains cumbersome. The diversity of XR hardware and software ecosystems directly impacts version management, as developing and testing experiences on heterogeneous setups (varying immersion, interaction capabilities) is complex and time-consuming. This is amplified by low robustness to OS, hardware, and 3D engine evolutions.

Research and Innovation Objectives:

• Investigate and design expressive, easy-to-use versioning tools for collaborative XR 3D scene editing. Further research is needed for XR scenes to converge towards robust versioning standards and integrated tools
• Explore tools and methods to perform authoring while experiencing VW on target devices and setups, from mobile (smartphones, tablets, headsets) to immersive spaces

2c.4 Integrating Artificial Intelligence into Design Workflows

Challenges and opportunities: Leveraging AI to significantly enhance and simplify the authoring of XR experiences. AI promises to support human-AI collaborative 3D scene editing, automate operations, and guide creators. This integration aims to streamline complex workflows, improve content quality, and ultimately make authoring solutions more accessible and powerful for a wider range of creators.

Problem Definition and Research Gap: AI offers great promise for supporting XR authoring but raises challenges concerning content generation and integration into existing solutions. Firstly, there's a lack of professional tools for guiding or specializing generative models, making it difficult to ensure the quality, coherence, and performance of AI-generated content (e.g., UIs, 3D models, code) and to validate assets against XR-specific constraints. Secondly, integrating AI and automatically generated assets into demanding XR pipelines is challenging. Current authoring solutions are already complex and target specialized developers; adding AI must simplify usage, not add complexity. Clear workflows and user interfaces are needed for seamless human-AI collaboration, allowing creators to steer and personalize AI outputs. Furthermore, rapidly evolving LLM capabilities complicate versioning, and AI model integration must comply with evolving ethical standards, privacy regulations (e.g., GDPR, AI Act), and transparency requirements, including equitable wealth distribution from AI-generated content.

Research and Innovation Objectives:

• Investigate tools and methods to inspect the quality, coherence, and performance of AI-generated content for XR scenes
• Explore human-AI collaboration approaches and techniques to support co-editing of VW content
• Facilitate the integration of AI agents and customizable models into authoring solution pipelines
• Develop standardized evaluation frameworks and best practices for AI-generated assets in XR, covering metrics like rendering efficiency, usability, and inclusivity
• Design systems for transparent versioning and traceability of AI contributions within collaborative workflows

2c.5 Accessibility of Solutions, No-Code Approaches and Creator Empowerment

Challenges and opportunities: Empowering a wider range of VW creators, especially non-experts, to contribute effectively to design and development, including for demanding XR experiences in uncontrolled environments. This addresses the high technical complexity and steep learning curves of current authoring solutions. Low-code to no-code approaches aim to democratise VW creation, essential given the diverse digital expertise levels.

Problem Definition and Research Gap: Current authoring solutions are too technically complex and have steep learning curves, preventing non-expert users from efficiently contributing. While approaches like authoring by demonstration, bodystorming with tangibles, and gesture-based 3D terrain editing exist, there are no unified solutions that cover all aspects of 3D virtual scene editing in an expressive yet accessible manner. Existing interaction techniques and tools are often limited to specific tasks and simple operations.

Research and Innovation Objectives:

• Expand existing accessible authoring tools and methods by investigating innovative low-to-no-code approaches for designing VW experiences, including authoring by demonstration and tangibles
• Investigate AI's potential to simplify workflows, automate operations, and provide guidance to creators, also in complex environments

2c.6 Real-Time Collaboration on Multiple Supports

Challenges and opportunities: Facilitating efficient real-time collaboration among diverse users (developers, artists, designers) for designing VW. This addresses the multidisciplinary nature of XR authoring, which demands flexible collaboration across various locations, times, and user groups. Real-time features would significantly shorten lengthy design processes, allowing for easier input, iteration, and faster convergence on refined designs.

Problem Definition and Research Gap: The multidisciplinary nature of XR authoring requires real-time collaboration tools that support diverse situations (co-located, hybrid, remote; asynchronous to synchronous; varying participant numbers, roles, and expertise). Current tools lack features for effectively tracking interactions, supporting group-awareness, providing fine control over private/public spaces, and facilitating 3D scene versioning. VW experiences also impose unique collaboration constraints due to the wide range of XR displays and devices. Creators use and test experiences on heterogeneous setups (tablets, smartphones, XR headsets, CAVE systems), whose differing capabilities (immersion, bulkiness, interaction modalities) create a significant barrier to fluid cross-device collaboration. While some platforms explore device-agnostic, multi-user environments, achieving truly fluid, cross-device collaboration requires deeper technical interoperability and thoughtful design of collaborative workflows, user interfaces, and interaction metaphors.

Research and Innovation Objectives:

• Investigate tools and methods for real-time collaboration in VW scene design. These should support diverse situations (locations, times, user groups), improve group-awareness, and manage private vs. public versions, leveraging social/cultural differences
• Explore innovative mechanisms to improve cross-device creator collaboration, including tailoring VW experiences to heterogeneous XR devices for both off-site and in-situ workflows. Cloud-based computing may be a promising direction
• Focus on solutions that integrate seamlessly into existing design and collaboration environments, enabling iterative, inclusive workflows for teams of varying expertise

2c.7 Interoperability of Experiences and Authoring Solutions

Challenges and opportunities: Enabling seamless interaction and data exchange between different VW experiences and authoring tools. This is crucial for overcoming the current fragmentation of the immersive ecosystem, which leads to lost efficiency, complicates creators' work, and hinders talent mobility. Achieving interoperability is a prerequisite for developing a coherent, sustainable, and collaborative ecosystem, especially for integrating diverse multisource content.

Problem Definition and Research Gap: The current XR authoring landscape is highly fragmented, with multiple creation environments, diverse programming languages, and incompatible proprietary formats (e.g. for 3D, IoT, BIM data). This results in lost collective efficiency, complicated workflows, and limited talent mobility. The low adoption of universal standards like OpenXR and glTF severely restricts interoperability between platforms and tools, hindering collaboration and multisource content integration. The growing diversity of multisource content (e.g., external information systems, real-time streams) to be integrated into experiences, coupled with co-creation by actors with varied expertise and device usage, further complicates interaction and demands unifying authoring approaches. Major technology players often impede interoperability to maintain control over content distribution and quality within their proprietary platforms. Furthermore, the absence of an independent authority to oversee interoperability standards increases the risk of lasting fragmentation.

Research and Innovation Objectives:

• Continue to apply and refine European standardisation strategies for XR to ensure adoption of open formats, guiding future norms towards sustainability, accessibility, and digital autonomy (e.g., Gaia-X project)
• Investigate and design a governance framework for an independent oversight authority responsible for ensuring adherence to interoperability standards by technology stakeholders

2c.8 Scalability and Performance

Challenges and opportunities: Ensuring that complex, collaborative, and interactive XR scenes can be rendered efficiently in real-time, regardless of device limitations or user scale. This addresses the challenge of resource-intensive computations required for high-quality VW, especially on user-worn XR devices with limited capabilities and for multi-user, worldwide VW applications. Optimised performance leads to better UX and enables larger-scale deployments.

Problem Definition and Research Gap: Rendering complex, collaborative, and interactive XR scenes in real-time requires heavy computation, yet many user-worn XR devices have limited technical capabilities. While tethered setups exist, many VW experiences need non-tethered devices for user mobility. Additionally, many XR experiences are multi-user oriented, with VW applications aiming for worldwide scale, involving significant data (heavy 3D models, point clouds, real-time environmental monitoring). These scalability and performance issues necessitate cloud-native and distributed-rendering solutions with appropriate network infrastructures. However, there are few commercially available solutions, and the absence of open-source EU-based authoring or distribution solutions raises concerns about EU autonomy for creating and diffusing large VW experiences. Finally, there's a gap in training professionals on these scalability, performance, and optimisation topics; technology education needs to emphasize computing-cost-saving practices.

Research and Innovation Objectives:

• Design open, cloud-native, and distributed rendering approaches for XR experiences. These should allow high-quality, real-time rendering for multi-user XR scenes across diverse devices and interactions
• Investigate methods and tools to better ensure the training of VW creators (including non-developers) on performance, scalability, adaptability to various technical constraints and optimisation aspects of XR experiences

2c.9 Adaptation to End User Contexts

Challenges and opportunities: Ensuring VW experiences are accessible and inclusive by dynamically adapting to diverse user profiles, environmental conditions, and hardware. This allows creators to tailor experiences for specific populations (e.g., impairments, varying digital skills) and environments, overcoming the current challenges of high project costs and reliance on scarce expert knowledge. Providing adaptable solutions improves UX and broadens XR adoption.

Problem Definition and Research Gap: Adapting VW experiences to target audiences involves tailoring many parameters related to user profiles (e.g., impairment, physical, mental, and digital skills), environmental conditions, and available hardware possibilities and limitations. While crucial for accessibility and inclusivity, these adaptations remain a challenge for professionals due to project costs and the need for specialized knowledge. Providing relevant adaptations often requires expert knowledge that is difficult for developers to acquire and is not adequately reflected in standard usability practices. Therefore, a significant research challenge lies in developing guidelines and solutions to facilitate and/or automate dynamic adaptations of VW interfaces, content, interaction, and modalities.

Research and Innovation Objectives:

• Investigate new guidelines, tools, and methods to dynamically adapt VW experiences, allowing creators to easily perform context-aware adaptations and design accessible applications
• Explore innovative ways to detect user and environmental conditions while preserving end-users' privacy and confidentiality, respecting EU laws

---

Additional Research Topic

2b.12 Understanding of Human Perception

Challenges and opportunities: A deep understanding of human perception is essential for developing intuitive and natural interactions in immersive environments. Neuroscientific insights into visual attention, multisensory integration, and cognitive load can inform how virtual environments should be designed to align with users' perceptual expectations and limitations. For instance, findings on motion sensitivity and spatial disorientation can help reduce VR-induced fatigue or cybersickness. Research into neuroplasticity and sensory substitution also opens pathways for making immersive systems more inclusive, enabling interaction for individuals with sensory impairments.

Problem Definition and Research Gap: While devices providing access to VW appear to have a catalytic role in human experiences and interactions, the human brain can easily adapt to current interfaces. Most people can achieve full immersion in a simulated 3D environment, even without specialized equipment, simply through screen-based graphic design. This human ability, however, is still not fully understood. Many factors contribute to the sensation of presence in VW, linked to internal brain processing by our different sensory modalities. This represents a significant research gap in understanding the complex interplay between human cognition and immersive technology.

Research and Innovation Objectives:

• Improve understanding of human perception within VW through neuroscience knowledge and user studies
• Gain a better understanding of the balance between the sensory modalities
• Enhance understanding of affective computing, which enables immersive systems to detect and respond to users' emotional states via modalities such as facial expression analysis, voice tone, and physiological data
• Integrate emotion-aware mechanisms into XR to enhance engagement, support mental health interventions, and personalise training scenarios
• Explore the role of neural synchrony, the alignment of brain activity between people, in enhancing group coordination and mutual understanding within virtual environments

---

Strategic Recommendations

For Industry and Developers

1. Invest in European authoring solutions: Support the development of open-source, XR-dedicated European authoring and distribution platforms to reduce dependency on non-EU ecosystems
2. Prioritise accessibility: Develop low-code/no-code authoring tools that democratize content creation and enable broader participation
3. Integrate AI responsibly: Leverage AI to simplify workflows while ensuring quality, coherence, and compliance with ethical standards and privacy regulations
4. Enable collaboration: Implement real-time, cross-device collaboration features that support multidisciplinary teams across diverse locations and expertise levels

For Research Institutions

1. Develop evaluation frameworks: Create standardized metrics, KPIs, and KVIs for assessing XR experience quality, utility, usability, and impact
2. Advance versioning systems: Research robust, XR-tailored version control and collaborative editing solutions that support diverse user profiles
3. Study human perception: Conduct neuroscientific research on multisensory integration, cognitive load, and presence to inform better experience design
4. Foster experimentation: Support R&D in experiential research, rapid prototyping, and iterative design methodologies

For Policymakers

1. Build European autonomy: Fund initiatives to develop sovereign authoring tools, distribution platforms, and technical communities independent of non-EU providers
2. Support creator ecosystems: Conduct studies on labour conditions and implement policies promoting equitable, sustainable creative environments
3. Establish training programs: Invest in multidisciplinary, accessible education that addresses the sector's rapid evolution and diverse skill requirements
4. Promote interoperability: Enforce adoption of open standards (OpenXR, glTF) and create governance frameworks for independent oversight

For Educators and Training Providers

1. Develop hybrid curricula: Create interdisciplinary programs combining technical skills (3D modelling, real-time programming) with creative disciplines (storytelling, UX design, cognitive psychology)
2. Bridge industry gaps: Establish partnerships between academic institutions and sectors (healthcare, education, retail) to align training with real-world needs
3. Enable continuous learning: Provide scalable, practice-oriented training models that adapt to rapidly evolving immersive tools and workflows

Cross-References

• 2a. Visualisation, Sensing, Devices and Immersion: For technical foundations of immersive experiences
• 2b. Real-time User Interaction: For UX design principles and interaction models
• 2d. Standardisation and Interoperability: For open standards enabling cross-platform authoring
• 2e. Digital Twins, Assets, and People: For asset management and digital identity systems
• 2f. Applied Artificial Intelligence: For AI-driven authoring assistance and content generation
• 3c. Economics and Innovation: For business models supporting sustainable creator ecosystems
• 3e. Trust and Human Oversight: For ethical AI integration and privacy-preserving authoring