About GuestXR PROJECT

AI agent addressing conflicts and unlocking interaction in extended reality environments

The GuestXR project develops an immersive virtual social space anchored in extended reality based on examining individual and group behaviour by drawing on existing theoretical models from the neuroscience and social psychology standpoint.

Thus, the GuestXR solution features a machine learning agent designed to unlock interactions between participants and help them achieve their goals, whilst helping users address conflicts in virtual environments.

The project technology was developed under the “Ethics by Design” approach. This entailed considering any potential ethical issues which may come up when using Artificial Intelligence such as respect for human autonomy, prevention of harm, fairness and explicability.

Key data

  • Start-end date: 1 January 2022 – 31 December 2025
  • Duration: 48 months
  • Funded under Horizon 2020 programme H2020-FETPROACT-2020-2
  • Overall budget: 4.499.519,76 €
  • Reference number: 101017884

OUR VISION

We developed a socially interactive multisensory platform system that uses extended reality, virtual and augmented reality, as the medium to bring people together for immersive, synchronous face-to-face interaction with positive social outcomes

PROJECT PHASES

From an idea to demonstrating the technology in relevant environment

1.

Development of the Machine Learning Agent

Application of deep Reinforcement Learning (RL) novel methods to train The Guest to obtain social intelligence, so that it can take specific actions that foster pro-social behaviour and social goals, in the context of Extended Reality applications incorporating ethics-by-design methodology.

2.

Social and affective modelling

Definition of agent psychological riles of behaviour, development of social behavioural simulations and computational description of body posture and movement features, as well as investigate how individuals’ expressions affect social interaction dynamics.

3.

Integration into experimental studies

Creation of the overall architecture of the AR and VR systems and development of the multiple representations of The Guest and its actions, as well as self-representation through embodiment in AR and of remote participants.

4.

Multisensory displays

Investigate and develop multimodal rendering such as haptics and research on multisensory integration and provide setups and guidelines for the project use cases.

5.

Use cases development

System validation through different use cases, including conflict situations or involving participants with communication problems.

THE GUEST

Helping people achieve their meeting goals

The Guest is a machine learning system to be devised as an agent that can examine participants’ individual and group behaviour by drawing on existing theoretical models from the neuroscience and social psychology standpoint.

This agent is capable of observing behaviour and detecting exclusion among the participants, and in this way, it intervenes to facilitate interaction, reduce conflict, and foster empathy, thus helping the group achieve their meeting’s objectives.

Therefore, the Guest is able to perform a range of multisensory actions, using, for instance, visual or auditory features to create particular states of mind and prompt a participant to share their views. 

Research beyond the state of the art

Ground-breaking research and development leading to novel social interactive systems mediated by a Machine Learning agent

Machine intelligence

Reinforcement Learning and algorithms to start testing The Guest in simulations, and then gradually transfer the agent to increasingly complex social scenarios with real humans, both in AR and VR.

Social rules processes

Computational simulations involving ABMs to allow pre-training, testing and adjustment of social rules that will define how to influence social interactions through GuestXR.

Real-time communication

Development of an automatic decoding system for emotions’ recognition from whole-body expressions in realistic situations based on computational movement feature analysis in combination with brain and body measures.

Goal-driven socially shared VR and AR

The research focused on how the interaction between people in VR and AR environments can be positively enhanced. This represents a first-time major proof of concept that a Machine Learning /AI system can positively enhance virtual meetings.

Multisensory integration

Establishment of new kinds of human-to-human communication to deliver tactile sensory inputs by combining audition with vision and haptics, as well as the sonification of visual movements to create an enhanced sensing.