In this page you will be able to find the scientific publications related to GuestXR project published by partners.
Resources
Scientific publications
Audio technology for improving social interaction in extended reality
Frontiers in Virtual Reality
by Joanna Luberadzka, Enric Gusó, Umuy Sayin & Adan Garriga
ABSTRACT
In recent years, extended reality (XR) has gained interest as a platform for human communication, with the emergence of the “Metaverse” promising to reshape social interactions. At the same time, concerns about harmful behavior and criminal activities in virtual environments have increased. This paper explores the potential of technology to support social harmony within XR, focusing specifically on audio aspects. We introduce the concept of acoustic coherence and discuss why it is crucial for smooth interaction. We further explain the challenges of speech communication in XR, including noise and reverberation, and review sound processing methods to enhance the auditory experience. We also comment on the potential of using virtual reality as a tool for the development and evaluation of audio algorithms aimed at enhancing communication. Finally, we present the results of a pilot study comparing several audio enhancement techniques inside a virtual environment.
Warm regards: Influence of thermal haptic feedback during social interactions in VR
2024 IEEE International Symposium on Mixed and Augmented Reality
by Jeanne Hecquard, Justine Saint-Aubert, Ferran Argelaguet, Claudio Pacchierotti, Anatole Lécuyer & Marc J.-M. Macé
ABSTRACT
In this paper, we study how thermal haptic feedback can influence social interactions in virtual environments, with an emphasis on persuasion, focus, co-presence, and friendliness. Physical and social warmth have been repeatedly linked in psychological literature, which allows for speculations on the effect of thermal haptics on virtual social interactions. To that effect, we conducted a study on thermal feedback during simulated social interactions with a virtual agent. We tested three conditions: warm, cool, and neutral. Results showed that warm feedback positively influenced users’ perception of the agent and significantly enhanced persuasion and thermal comfort. Multiple users reported the agent feeling less ’robotic’ and more ’human’ during the warm condition. Moreover, multiple studies have previously shown the potential of vibrotactile feedback for social interactions. A second study thus evaluated the combination of warmth and vibrations for social interactions. The study included the same protocol and three similar conditions: warmth, vibrations, and warm vibrations. Warmth was perceived as more friendly, while warm vibrations heightened the agent’s virtual presence and persuasion. These results encourage the study of thermal haptics to support positive social interactions. Moreover, they suggest that some haptic feedback are more suited to certain types of social interactions and communication than others.
Harnessing Immersive Technologies: Modern Perspectives on Mental Health Support
2024 IEEE International Symposium on Mixed and Augmented Reality
by Esen Küçüktütüncü & Lisa Izzaouzi
ABSTRACT
Although psychological, social, and emotional well-being, collectively known as mental health, has not always been a primary focus, it has become increasingly important in recent years. With mental health being valued more and digital transformation becoming prevalent across various fields, there have been notable advancements in integrating mental health into the digital era. In this paper, we offer an overview of the digitization of mental health in recent years, more specifically through immersive technologies. We explore the benefits and challenges of integrating these technologies into clinical practice and highlight tools that could revolutionize the field. We propose directions for further research and emphasize that while digital advancements are valuable, immersive technologies should be used to enhance rather than replace traditional practices, always preferring a human-centered approach.
Enhancing interoceptive sensibility through exteroceptive–interoceptive sensory substitution
Scientific reports
by Oran Goral, Iddo Yehoshua Wald, Amber Maimon, Adi Snir, Yulia Golland, Aviva Goral & Amir Amedi
ABSTRACT
Exploring a novel approach to mental health technology, this study illuminates the intricate interplay between exteroception (the perception of the external world), and interoception (the perception of the internal world). Drawing on principles of sensory substitution, we investigated how interoceptive signals, particularly respiration, could be conveyed through exteroceptive modalities, namely vision and hearing. To this end, we developed a unique, immersive multisensory environment that translates respiratory signals in real-time into dynamic visual and auditory stimuli. The system was evaluated by employing a battery of various psychological assessments, with the findings indicating a significant increase in participants’ interoceptive sensibility and an enhancement of the state of flow, signifying immersive and positive engagement with the experience. Furthermore, a correlation between these two variables emerged, revealing a bidirectional enhancement between the state of flow and interoceptive sensibility. Our research is the first to present a sensory substitution approach for substituting between interoceptive and exteroceptive senses, and specifically as a transformative method for mental health interventions, paving the way for future research.
Interoceptive training with real-time haptic versus visualheartbeat feedback
Psychophisiology
by Olga Dobrushina, Yossi Tamim, Iddo Yehoshua Wald, Amber Maimon, Amir Amedi
ABSTRACT
The perception of signals from within the body, known as interoception, is in-creasingly recognized as a prerequisite for physical and mental health. This studyis dedicated to the development of effective technological approaches for enhanc-ing interoceptive abilities. We provide evidence of the effectiveness and practi-cal feasibility of a novel real-time haptic heartbeat supplementation technologycombining principles of biofeedback and sensory augmentation. In a randomizedcontrolled study, we applied the developed naturalistic haptic feedback on agroup of 30 adults, while another group of 30 adults received more traditionalreal-time visual heartbeat feedback. A single session of haptic, but not visualheartbeat feedback resulted in increased interoceptive accuracy and confidence,as measured by the heart rate discrimination task, and in a shift of attention to-ward the body. Participants rated the developed technology as more helpful andpleasant than the visual feedback, thus indicating high user satisfaction. Thestudy highlights the importance of matching sensory characteristics of the feed-back provided to the natural bodily prototype. Our work suggests that real-timehaptic feedback might be a superior approach for strengthening the mind–bodyconnection in interventions for physical and mental health.
Localizing 3D motion through the fingertips: Following in the footsteps of elephants
iScience
by Adi Snir, Katarzyna Cieśla, Gizem Ozdemir, Rotem Vekslar & Amir Amedi
ABSTRACT
Highly compromised auditory spatial perception in aided congenitally hearing-impaired and rapid improvement with tactile technology
iScience
by Adi Snir, Katarzyna Cieśla, Rotem Vekslar & Amir Amedi
ABSTRACT
Motivational Interviewing Transcripts Annotated with Global Scores
LREC-COLING 2024
by Ben Cohen, Moreah Zisquit, Stav Yosef, Anat Klomek Brunstein, Doron Friedman & Kfir Bar
ABSTRACT
Assessing Motivational Interviewing Sessions with AI-Generated Patient Simulations
CLPsych 2024
by Stav Yosef, Moreah Zisquit, Ben Cohen, Anat Brunstein Klomek, Kfir Bar & Doron Friedman
ABSTRACT
There is growing interest in utilizing large language models (LLMs) in the field of mental health, and this goes as far as suggesting automated LLM-based therapists. Evaluating such generative models in therapy sessions is essential, yet remains an ongoing and complex challenge. We suggest a novel approach: an LLMbased digital patient platform which generates digital patients that can engage in a text-based conversation with either automated or human therapists. Moreover, we show that LLMs can be used to rate the quality of such sessions by completing questionnaires originally designed for human patients. We demonstrate that the ratings are both statistically reliable and valid, indicating that they are consistent and capable of distinguishing among three levels of therapist expertise. In the present study, we focus on motivational interviewing, but we suggest that this platform can be adapted to facilitate other types of therapies. We plan to publish the digital patient platform and make it available to the research community, with the hope of contributing to the standardization of evaluating automated therapists.
Rapid plasticity in the ventral visual stream elicited by a newly learnt auditory script in congenitally blind adults
Neuropsychologia
by Roni Arbel, Benedetta Heimler & Amir Amedi
ABSTRACT
Accumulating evidence in the last decades has given rise to a new theory of brain organization, positing that cortical regions are recruited for specific tasks irrespective of the sensory modality via which information is channeled. For instance, the visual reading network has been shown to be recruited for reading via the tactile Braille code in congenitally blind adults. Yet, how rapidly non-typical sensory input modulates activity in typically visual regions is yet to be explored. To this aim, we developed a novel reading orthography, termed OVAL, enabling congenitally blind adults to quickly acquire reading via the auditory modality. OVAL uses the EyeMusic, a visual-to-auditory sensory-substitution-device (SSD) to transform visually presented letters optimized for auditory transformation into sound. Using fMRI, we show modulation in the right ventral visual stream following 2-h of same-day training. Crucially, following more extensive training (i.e., ∼12 h) we show that OVAL reading recruits the left ventral visual stream including the location of the Visual Word Form Area, a key graphene-responsive region within the visual reading network. Our results show that while after 2 h of SSD training we can already observe the recruitment of the deprived ventral visual stream by auditory stimuli, computation-selective cross-modal recruitment requires longer training to establish.
Loss of action-related function and connectivity in the blind extrastriate body area
Frontiers in Neuroscience
by Or Yizhar, Zohar Tal & Amir Amedi
ABSTRACT
The Extrastriate Body Area (EBA) participates in the visual perception and motor actions of body parts. We recently showed that EBA’s perceptual function develops independently of visual experience, responding to stimuli with body-part information in a supramodal fashion. However, it is still unclear if the EBA similarly maintains its action-related function. Here, we used fMRI to study motor-evoked responses and connectivity patterns in the congenitally blind brain. We found that, unlike the case of perception, EBA does not develop an action-related response without visual experience. In addition, we show that congenital blindness alters EBA’s connectivity profile in a counterintuitive way—functional connectivity with sensorimotor cortices dramatically decreases, whereas connectivity with perception-related visual occipital cortices remains high. To the best of our knowledge, we show for the first time that action-related functions and connectivity in the visual cortex could be contingent on visuomotor experience. We further discuss the role of the EBA within the context of visuomotor control and predictive coding theory.
Early threat perception is independent of later cognitive and behavioral control. A virtual reality-EEG-ECG study
Cerebral Cortex
by Juanzhi Lu, Selma K. Kemmerer, Lars Riecke & Beatrice de Gelder
ABSTRACT
Research on social threat has shown influences of various factors, such as agent characteristics, proximity, and social interaction on social threat perception. An important, yet understudied aspect of threat exposure concerns the ability to exert control over the threat and its implications for threat perception. In this study, we used a virtual reality (VR) environment showing an approaching avatar that was either angry (threatening body expression) or neutral (neutral body expression) and informed participants to stop avatars from coming closer under five levels of control success (0, 25, 50, 75, or 100%) when they felt uncomfortable. Behavioral results revealed that social threat triggered faster reactions at a greater virtual distance from the participant than the neutral avatar. Event-related potentials (ERPs) revealed that the angry avatar elicited a larger N170/vertex positive potential (VPP) and a smaller N3 than the neutral avatar. The 100% control condition elicited a larger late positive potential (LPP) than the 75% control condition. In addition, we observed enhanced theta power and accelerated heart rate for the angry avatar vs. neutral avatar, suggesting that these measures index threat perception. Our results indicate that perception of social threat takes place in early to middle cortical processing stages, and control ability is associated with cognitive evaluation in middle to late stages.
Ultrahigh Field fMRI Reveals Different Roles of the Temporal and Frontoparietal Cortices in Subjective Awareness
Journal of Neuroscience
by Marta Poyo, Minye Zhan & Beatrice de Gelder
ABSTRACT
A central question in consciousness theories is whether one is dealing with a dichotomous (“all-or-none”) or a gradual phenomenon. In this 7T fMRI study, we investigated whether dichotomy or gradualness in fact depends on the brain region associated with perceptual awareness reports. Both male and female human subjects performed an emotion discrimination task (fear vs neutral bodies) presented under continuous flash suppression with trial-based perceptual awareness measures. Behaviorally, recognition sensitivity increased linearly with increased stimuli awareness and was at chance level during perceptual unawareness. Physiologically, threat stimuli triggered a slower heart rate than neutral ones during “almost clear” stimulus experience, indicating freezing behavior. Brain results showed that activity in the occipitotemporal, parietal, and frontal regions as well as in the amygdala increased with increased stimulus awareness while early visual areas showed the opposite pattern. The relationship between temporal area activity and perceptual awareness best fitted a gradual model while the activity in frontoparietal areas fitted a dichotomous model. Furthermore, our findings illustrate that specific experimental decisions, such as stimulus type or the approach used to evaluate awareness, play pivotal roles in consciousness studies and warrant careful consideration.
Theta activity discriminates high-level, species-specific body processes
Imaging Neuroscience
by Jane Chesley, Lars Riecke, Juanzhi Lu, Rufin Vogels & Beatrice de Gelder
ABSTRACT
Among social stimuli that trigger rapid reactions, body images occupy a prominent place. Given that bodies carry information about other agents’ intentions, actions and emotional expressions, a foundational question concerns the neural basis of body processing. Previous fMRI studies have investigated this but were not yet able to clarify the time course and its functional significance. The present EEG study investigated the role of slow oscillatory cortical activity in body processing and species-specificity. Human participants viewed naturalistic images of human and monkey bodies, faces, and objects, along with mosaic-scrambled versions to control for low-level visual features. Analysis of event-related theta power (4 – 7 Hz) combined with data-driven methods revealed a strong, body-evoked neural response that is specific to human bodies and likely originates from a widespread cortical region during a time window of 150 – 550 ms after the onset of the body image. Our results corroborate recent research proposing a widespread, species-specific cortical network of human body processing. We submit that this network may play an essential role in linking body processes to movement intentions.
Differential contributions of body form, motion, and temporal information to subjective action understanding in naturalistic stimuli
Frontiers in Integrative Neuroscience
by Vojtěch Smekal; Marta Poyo Solanas; Evelyne I. C. Fraats & Beatrice de Gelder
ABSTRACT
Introduction: We investigated the factors underlying naturalistic action recognition and understanding, as well as the errors occurring during recognition failures.
Methods: Participants saw full-light stimuli of ten different whole-body actions presented in three different conditions: as normal videos, as videos with the temporal order of the frames scrambled, and as single static representative frames. After each stimulus presentation participants completed one of two tasks—a forced choice task where they were given the ten potential action labels as options, or a free description task, where they could describe the action performed in each stimulus in their own words.
Results: While generally, a combination of form, motion, and temporal information led to the highest action understanding, for some actions form information was sufficient and adding motion and temporal information did not increase recognition accuracy. We also analyzed errors in action recognition and found primarily two different types.
Discussion: One type of error was on the semantic level, while the other consisted of reverting to the kinematic level of body part processing without any attribution of semantics. We elaborate on these results in the context of naturalistic action perception.
Studying the influence of contact force on thermal perception at the fingertip
Eurohaptics 2024
by Danilo Troisi, Jeanne Hecquard, Ferran Argelaguet, Justine Saint-Aubert, Marc J.-M. Macé, Anatole Lécuyer, Claudio Pacchierotti & Monica Malvezzi
ABSTRACT
This paper investigates the influence of contact force applied to the human’s fingertip on the perception of hot and cold temperatures, studying how variations in contact force may affect the sensitivity of cutaneous thermoreceptors or their interpretation. A psychophysical experiment involved 18 participants exposed to cold (20 °C) and hot (38 °C) thermal stimuli at varying contact forces, ranging from gentle (0.5 N) to firm (3.5 N) touch. Results show a tendency to overestimate hot temperatures (hot feels hotter than it really is) and underestimate cold temperatures (cold feels colder than it really is) as the contact force increases. This result might be linked to the increase in the fingertip contact area that occurs as the contact force between the fingertip and the plate delivering the stimuli grows.
Cool Me Down: Effects of Thermal Feedback on Cognitive Stress in Virtual Reality
Eurohaptics 2024
by Vincent Philippe, Jeanne Hecquard, Emilie Hummel, Ferran Argelaguet, Marc J.-M. Macé, Valérie Gouranton, Claudio Pacchierotti, Anatole Lécuyer & Justine Saint-Aubert
ABSTRACT
We investigate the influence of thermal haptic feedback on stress during a cognitive task in virtual reality. We hypothesized that cool feedback would help reduce stress in such a task where users are actively engaged. We designed a haptic system using Peltier cells to deliver thermal feedback to the left and right trapezius muscles. A user study was conducted on 36 participants to investigate the influence of different temperatures (cool, warm, neutral) on users’ stress during mental arithmetic tasks. Results show that the impact of the thermal feedback depends on the participant’s temperature preference. Interestingly, a subset of participants (36%) felt less stressed with cool feedback than with neutral feedback, but had similar performance levels, and expressed a preference for the cool condition. Emotional arousal also tended to be lower with cool feedback for these participants. This suggests that cool thermal feedback has the potential to induce relaxation in cognitive tasks. Taken together, our results pave the way for further experimentation on the influence of thermal feedback.
"Let's ask what AI thinks then!": Using LLMs for Collaborative Problem-Solving in Virtual Environments
2024 IEEE International Symposium on Mixed and Augmented Reality
by Esen Küçüktütüncü & Lisa Izzaouzi
ABSTRACT
Harnessing Immersive Technologies: Modern Perspectives on Mental Health Support
2024 IEEE International Symposium on Mixed and Augmented Reality
by Esen Küçüktütüncü & Lisa Izzaouzi
ABSTRACT
Although psychological, social, and emotional well-being, collectively known as mental health, has not always been a primary focus, it has become increasingly important in recent years. With mental health being valued more and digital transformation becoming prevalent across various fields, there have been notable advancements in integrating mental health into the digital era. In this paper, we offer an overview of the digitization of mental health in recent years, more specifically through immersive technologies. We explore the benefits and challenges of integrating these technologies into clinical practice and highlight tools that could revolutionize the field. We propose directions for further research and emphasize that while digital advancements are valuable, immersive technologies should be used to enhance rather than replace traditional practices, always preferring a human-centered approach.
Enhancing interoceptive sensibility through exteroceptive–interoceptive sensory substitution
Scientific reports
by Oran Goral, Iddo Yehoshua Wald, Amber Maimon, Adi Snir, Yulia Golland, Aviva Goral & Amir Amedi
ABSTRACT
Exploring a novel approach to mental health technology, this study illuminates the intricate interplay between exteroception (the perception of the external world), and interoception (the perception of the internal world). Drawing on principles of sensory substitution, we investigated how interoceptive signals, particularly respiration, could be conveyed through exteroceptive modalities, namely vision and hearing. To this end, we developed a unique, immersive multisensory environment that translates respiratory signals in real-time into dynamic visual and auditory stimuli. The system was evaluated by employing a battery of various psychological assessments, with the findings indicating a significant increase in participants’ interoceptive sensibility and an enhancement of the state of flow, signifying immersive and positive engagement with the experience. Furthermore, a correlation between these two variables emerged, revealing a bidirectional enhancement between the state of flow and interoceptive sensibility. Our research is the first to present a sensory substitution approach for substituting between interoceptive and exteroceptive senses, and specifically as a transformative method for mental health interventions, paving the way for future research.
Interoceptive training with real-time haptic versus visualheartbeat feedback
Psychophisiology
by Olga Dobrushina, Yossi Tamim, Iddo Yehoshua Wald, Amber Maimon, Amir Amedi
ABSTRACT
The perception of signals from within the body, known as interoception, is in-creasingly recognized as a prerequisite for physical and mental health. This studyis dedicated to the development of effective technological approaches for enhanc-ing interoceptive abilities. We provide evidence of the effectiveness and practi-cal feasibility of a novel real-time haptic heartbeat supplementation technologycombining principles of biofeedback and sensory augmentation. In a randomizedcontrolled study, we applied the developed naturalistic haptic feedback on agroup of 30 adults, while another group of 30 adults received more traditionalreal-time visual heartbeat feedback. A single session of haptic, but not visualheartbeat feedback resulted in increased interoceptive accuracy and confidence,as measured by the heart rate discrimination task, and in a shift of attention to-ward the body. Participants rated the developed technology as more helpful andpleasant than the visual feedback, thus indicating high user satisfaction. Thestudy highlights the importance of matching sensory characteristics of the feed-back provided to the natural bodily prototype. Our work suggests that real-timehaptic feedback might be a superior approach for strengthening the mind–bodyconnection in interventions for physical and mental health.
Localizing 3D motion through the fingertips: Following in the footsteps of elephants
iScience
by Adi Snir, Katarzyna Cieśla, Gizem Ozdemir, Rotem Vekslar & Amir Amedi
ABSTRACT
Highly compromised auditory spatial perception in aided congenitally hearing-impaired and rapid improvement with tactile technology
iScience
by Adi Snir, Katarzyna Cieśla, Rotem Vekslar & Amir Amedi
ABSTRACT
Motivational Interviewing Transcripts Annotated with Global Scores
LREC-COLING 2024
by Ben Cohen, Moreah Zisquit, Stav Yosef, Anat Klomek Brunstein, Doron Friedman & Kfir Bar
ABSTRACT
Assessing Motivational Interviewing Sessions with AI-Generated Patient Simulations
CLPsych 2024
by Stav Yosef, Moreah Zisquit, Ben Cohen, Anat Brunstein Klomek, Kfir Bar & Doron Friedman
ABSTRACT
There is growing interest in utilizing large language models (LLMs) in the field of mental health, and this goes as far as suggesting automated LLM-based therapists. Evaluating such generative models in therapy sessions is essential, yet remains an ongoing and complex challenge. We suggest a novel approach: an LLMbased digital patient platform which generates digital patients that can engage in a text-based conversation with either automated or human therapists. Moreover, we show that LLMs can be used to rate the quality of such sessions by completing questionnaires originally designed for human patients. We demonstrate that the ratings are both statistically reliable and valid, indicating that they are consistent and capable of distinguishing among three levels of therapist expertise. In the present study, we focus on motivational interviewing, but we suggest that this platform can be adapted to facilitate other types of therapies. We plan to publish the digital patient platform and make it available to the research community, with the hope of contributing to the standardization of evaluating automated therapists.
Rapid plasticity in the ventral visual stream elicited by a newly learnt auditory script in congenitally blind adults
Neuropsychologia
by Roni Arbel, Benedetta Heimler & Amir Amedi
ABSTRACT
Accumulating evidence in the last decades has given rise to a new theory of brain organization, positing that cortical regions are recruited for specific tasks irrespective of the sensory modality via which information is channeled. For instance, the visual reading network has been shown to be recruited for reading via the tactile Braille code in congenitally blind adults. Yet, how rapidly non-typical sensory input modulates activity in typically visual regions is yet to be explored. To this aim, we developed a novel reading orthography, termed OVAL, enabling congenitally blind adults to quickly acquire reading via the auditory modality. OVAL uses the EyeMusic, a visual-to-auditory sensory-substitution-device (SSD) to transform visually presented letters optimized for auditory transformation into sound. Using fMRI, we show modulation in the right ventral visual stream following 2-h of same-day training. Crucially, following more extensive training (i.e., ∼12 h) we show that OVAL reading recruits the left ventral visual stream including the location of the Visual Word Form Area, a key graphene-responsive region within the visual reading network. Our results show that while after 2 h of SSD training we can already observe the recruitment of the deprived ventral visual stream by auditory stimuli, computation-selective cross-modal recruitment requires longer training to establish.
Loss of action-related function and connectivity in the blind extrastriate body area
Frontiers in Neuroscience
by Or Yizhar, Zohar Tal & Amir Amedi
ABSTRACT
The Extrastriate Body Area (EBA) participates in the visual perception and motor actions of body parts. We recently showed that EBA’s perceptual function develops independently of visual experience, responding to stimuli with body-part information in a supramodal fashion. However, it is still unclear if the EBA similarly maintains its action-related function. Here, we used fMRI to study motor-evoked responses and connectivity patterns in the congenitally blind brain. We found that, unlike the case of perception, EBA does not develop an action-related response without visual experience. In addition, we show that congenital blindness alters EBA’s connectivity profile in a counterintuitive way—functional connectivity with sensorimotor cortices dramatically decreases, whereas connectivity with perception-related visual occipital cortices remains high. To the best of our knowledge, we show for the first time that action-related functions and connectivity in the visual cortex could be contingent on visuomotor experience. We further discuss the role of the EBA within the context of visuomotor control and predictive coding theory.
Early threat perception is independent of later cognitive and behavioral control. A virtual reality-EEG-ECG study
Cerebral Cortex
by Juanzhi Lu, Selma K. Kemmerer, Lars Riecke & Beatrice de Gelder
ABSTRACT
Research on social threat has shown influences of various factors, such as agent characteristics, proximity, and social interaction on social threat perception. An important, yet understudied aspect of threat exposure concerns the ability to exert control over the threat and its implications for threat perception. In this study, we used a virtual reality (VR) environment showing an approaching avatar that was either angry (threatening body expression) or neutral (neutral body expression) and informed participants to stop avatars from coming closer under five levels of control success (0, 25, 50, 75, or 100%) when they felt uncomfortable. Behavioral results revealed that social threat triggered faster reactions at a greater virtual distance from the participant than the neutral avatar. Event-related potentials (ERPs) revealed that the angry avatar elicited a larger N170/vertex positive potential (VPP) and a smaller N3 than the neutral avatar. The 100% control condition elicited a larger late positive potential (LPP) than the 75% control condition. In addition, we observed enhanced theta power and accelerated heart rate for the angry avatar vs. neutral avatar, suggesting that these measures index threat perception. Our results indicate that perception of social threat takes place in early to middle cortical processing stages, and control ability is associated with cognitive evaluation in middle to late stages.
Ultrahigh Field fMRI Reveals Different Roles of the Temporal and Frontoparietal Cortices in Subjective Awareness
Journal of Neuroscience
by Marta Poyo, Minye Zhan & Beatrice de Gelder
ABSTRACT
A central question in consciousness theories is whether one is dealing with a dichotomous (“all-or-none”) or a gradual phenomenon. In this 7T fMRI study, we investigated whether dichotomy or gradualness in fact depends on the brain region associated with perceptual awareness reports. Both male and female human subjects performed an emotion discrimination task (fear vs neutral bodies) presented under continuous flash suppression with trial-based perceptual awareness measures. Behaviorally, recognition sensitivity increased linearly with increased stimuli awareness and was at chance level during perceptual unawareness. Physiologically, threat stimuli triggered a slower heart rate than neutral ones during “almost clear” stimulus experience, indicating freezing behavior. Brain results showed that activity in the occipitotemporal, parietal, and frontal regions as well as in the amygdala increased with increased stimulus awareness while early visual areas showed the opposite pattern. The relationship between temporal area activity and perceptual awareness best fitted a gradual model while the activity in frontoparietal areas fitted a dichotomous model. Furthermore, our findings illustrate that specific experimental decisions, such as stimulus type or the approach used to evaluate awareness, play pivotal roles in consciousness studies and warrant careful consideration.
Theta activity discriminates high-level, species-specific body processes
Imaging Neuroscience
by Jane Chesley, Lars Riecke, Juanzhi Lu, Rufin Vogels & Beatrice de Gelder
ABSTRACT
Among social stimuli that trigger rapid reactions, body images occupy a prominent place. Given that bodies carry information about other agents’ intentions, actions and emotional expressions, a foundational question concerns the neural basis of body processing. Previous fMRI studies have investigated this but were not yet able to clarify the time course and its functional significance. The present EEG study investigated the role of slow oscillatory cortical activity in body processing and species-specificity. Human participants viewed naturalistic images of human and monkey bodies, faces, and objects, along with mosaic-scrambled versions to control for low-level visual features. Analysis of event-related theta power (4 – 7 Hz) combined with data-driven methods revealed a strong, body-evoked neural response that is specific to human bodies and likely originates from a widespread cortical region during a time window of 150 – 550 ms after the onset of the body image. Our results corroborate recent research proposing a widespread, species-specific cortical network of human body processing. We submit that this network may play an essential role in linking body processes to movement intentions.
Differential contributions of body form, motion, and temporal information to subjective action understanding in naturalistic stimuli
Frontiers in Integrative Neuroscience
by Vojtěch Smekal; Marta Poyo Solanas; Evelyne I. C. Fraats & Beatrice de Gelder
ABSTRACT
Introduction: We investigated the factors underlying naturalistic action recognition and understanding, as well as the errors occurring during recognition failures.
Methods: Participants saw full-light stimuli of ten different whole-body actions presented in three different conditions: as normal videos, as videos with the temporal order of the frames scrambled, and as single static representative frames. After each stimulus presentation participants completed one of two tasks—a forced choice task where they were given the ten potential action labels as options, or a free description task, where they could describe the action performed in each stimulus in their own words.
Results: While generally, a combination of form, motion, and temporal information led to the highest action understanding, for some actions form information was sufficient and adding motion and temporal information did not increase recognition accuracy. We also analyzed errors in action recognition and found primarily two different types.
Discussion: One type of error was on the semantic level, while the other consisted of reverting to the kinematic level of body part processing without any attribution of semantics. We elaborate on these results in the context of naturalistic action perception.
Studying the influence of contact force on thermal perception at the fingertip
Eurohaptics 2024
by Danilo Troisi, Jeanne Hecquard, Ferran Argelaguet, Justine Saint-Aubert, Marc J.-M. Macé, Anatole Lécuyer, Claudio Pacchierotti & Monica Malvezzi
ABSTRACT
This paper investigates the influence of contact force applied to the human’s fingertip on the perception of hot and cold temperatures, studying how variations in contact force may affect the sensitivity of cutaneous thermoreceptors or their interpretation. A psychophysical experiment involved 18 participants exposed to cold (20 °C) and hot (38 °C) thermal stimuli at varying contact forces, ranging from gentle (0.5 N) to firm (3.5 N) touch. Results show a tendency to overestimate hot temperatures (hot feels hotter than it really is) and underestimate cold temperatures (cold feels colder than it really is) as the contact force increases. This result might be linked to the increase in the fingertip contact area that occurs as the contact force between the fingertip and the plate delivering the stimuli grows.
Cool Me Down: Effects of Thermal Feedback on Cognitive Stress in Virtual Reality
Eurohaptics 2024
by Vincent Philippe, Jeanne Hecquard, Emilie Hummel, Ferran Argelaguet, Marc J.-M. Macé, Valérie Gouranton, Claudio Pacchierotti, Anatole Lécuyer & Justine Saint-Aubert
ABSTRACT
We investigate the influence of thermal haptic feedback on stress during a cognitive task in virtual reality. We hypothesized that cool feedback would help reduce stress in such a task where users are actively engaged. We designed a haptic system using Peltier cells to deliver thermal feedback to the left and right trapezius muscles. A user study was conducted on 36 participants to investigate the influence of different temperatures (cool, warm, neutral) on users’ stress during mental arithmetic tasks. Results show that the impact of the thermal feedback depends on the participant’s temperature preference. Interestingly, a subset of participants (36%) felt less stressed with cool feedback than with neutral feedback, but had similar performance levels, and expressed a preference for the cool condition. Emotional arousal also tended to be lower with cool feedback for these participants. This suggests that cool thermal feedback has the potential to induce relaxation in cognitive tasks. Taken together, our results pave the way for further experimentation on the influence of thermal feedback.
Haptics in the Metaverse: Haptic feedback for Virtual, Augmented, Mixed, and eXtended Realities
IEEE Transactions on Haptics (ToH)
by Claudio Pacchierotti, Francesco Chinello, Konstantinos Koumaditis, Massimiliano Di Luca, Eyal Ofek & Orestis Georgiou
ABSTRACT
In the last few years, we have witnessed the rapid development of many innovative devices and original techniques for providing haptic sensations, e.g., using force feedback, mid-air interfaces, props and encounter-type devices, or exploiting perceptual phenomena with cross-modal effects such as pseudo-haptics. While increasingly immersive and realistic experiences have developed at a fast pace, the emergence of a “metaverse” proposes new use cases where prolonged utilisation and social interactions become more frequent and widespread. The metaverse definition is continuously evolving, however, for now it can been seen as a collective virtual shared space, created by the convergence of the physical and digital worlds, where users interact, socialize, and engage with each other through digital representations of themselves. This new direction in social interactions presents the haptics community with new challenges and opportunities. Indeed, as eXtended Reality (XR) technologies continue to gain traction in various fields, the role of haptic feedback and feedforward in enhancing such experiences have become increasingly evident. This growing significance is reflected in the proliferation of research efforts and the emergence of dedicated haptic designs, tools, and rendering techniques. Never before have tactile actuators found such proliferation in 3D applications, akin to that experienced in the early days of smartphones.
This special issue is dedicated to exploring the multifaceted realm of haptics tailored to XR environments, including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). The collection of studies presented here represents a diverse array of topics and approaches, each contributing to our fundamental understanding of haptics in the context of such immersive technologies. Some of the featured research found in this special issue delves into the theoretical foundations of haptics in XR, unveiling the complexities of human haptic perception, touch, and proprioception. Understanding how users perceive and interact in virtual environments is paramount to crafting truly immersive experiences that mimic the sense of touch and presence of the real world. In parallel, other contributions focus on the practical aspects of haptic interface design, exploring new methods, principles, and guidelines to create more effective and intuitive haptic interactions. These studies aim to strike a delicate balance between providing realistic touch sensations and ensuring comfortable, ergonomic, and user-friendly haptic devices. A significant portion of this special issue is dedicated to the advancements in haptic actuation and sensing technologies. From traditional vibrotactile actuators to more sophisticated systems like ultrasound, electrostatic, piezoelectric, and magnetic actuators, researchers have been striving to create more versatile, precise, and responsive haptic feedback mechanisms that align seamlessly with multimodal and multisensory virtual and augmented realities. Finally, as XR technologies find applications in a wide range of industries, this special issue showcases novel haptic approaches tailored to specific domains. Whether it be industrial training or artistic expression, these studies demonstrate the adaptability of haptic interfaces to diverse applications and contexts.
We believe that the research presented in this special issue will not only advance our knowledge of haptics in XR, but will also inspire further innovation and foster cross-disciplinary collaborations. As technology continues to push the boundaries of what is possible in the realm of virtual and augmented reality, haptic interfaces will play an increasingly pivotal role in shaping the future of immersive (social) experiences.
We have received many exceptional submissions for this special issue, indicating the research community’s profound enthusiasm for haptics in such a field. Our gratitude goes to the Editors in Chief, Domenico Prattichizzo, Seungmoon Choi, and Marcia K. O’Malley, for their support to the success of this topic. Additionally, we extend our heartfelt appreciation to the numerous reviewers who contributed highquality and timely reviews. Lastly, we would like to express our thanks to Shivam Sony, a member of the ToH editorial staff, for his precious assistance throughout the publication process. We hope that this collection of 14 research papers will serve as a valuable resource for researchers, practitioners, and enthusiasts, driving us all closer to a more haptically enriched virtual and augmented world.
Modality-specific brain representations during automatic processing of face, voice and body expressions
Frontiers in Neuroscience
by Maarten Vaessen, Kiki Van der Heijden & Beatrice de Gelder
ABSTRACT
A central question in affective science and one that is relevant for its clinical applications is how emotions provided by different stimuli are experienced and represented in the brain. Following the traditional view emotional signals are recognized with the help of emotion concepts that are typically used in descriptions of mental states and emotional experiences, irrespective of the sensory modality. This perspective motivated the search for abstract representations of emotions in the brain, shared across variations in stimulus type (face, body, voice) and sensory origin (visual, auditory). On the other hand, emotion signals like for example an aggressive gesture, trigger rapid automatic behavioral responses and this may take place before or independently of full abstract representation of the emotion. This pleads in favor specific emotion signals that may trigger rapid adaptative behavior only by mobilizing modality and stimulus specific brain representations without relying on higher order abstract emotion categories. To test this hypothesis, we presented participants with naturalistic dynamic emotion expressions of the face, the whole body, or the voice in a functional magnetic resonance (fMRI) study. To focus on automatic emotion processing and sidestep explicit concept-based emotion recognition, participants performed an unrelated target detection task presented in a different sensory modality than the stimulus. By using multivariate analyses to assess neural activity patterns in response to the different stimulus types, we reveal a stimulus category and modality specific brain organization of affective signals. Our findings are consistent with the notion that under ecological conditions emotion expressions of the face, body and voice may have different functional roles in triggering rapid adaptive behavior, even if when viewed from an abstract conceptual vantage point, they may all exemplify the same emotion. This has implications for a neuroethologically grounded emotion research program that should start from detailed behavioral observations of how face, body, and voice expressions function in naturalistic contexts.
The Making of a Newspaper Interview in Virtual Reality: Realistic Avatars, Philosophy, and Sushi
IEEE Computer Graphics and Applications
by Ramon Oliva, Alejandro Beacco, Jaime Gallego, Raul Gallego & Mel Slater
ABSTRACT
VR United is a virtual reality application that we have developed to support multiple people simultaneously interacting in the same environment. Each person is represented with a virtual body that looks like themselves. Such immersive shared environments have existed and been the subject of research for the past 30 years. Here, we demonstrate how VR United meets criteria for successful interaction, where a journalist from the Financial Times in London interviewed a professor in New York for two hours. The virtual location of the interview was a restaurant, in line with the series of interviews published as “Lunch with the FT.” We show how the interview was successful, as a substitute for a physically present one. The article based on the interview was published in the Financial Times as normal for the series. We finally consider the future development of such systems, including some implications for immersive journalism.
An objective evaluation of Hearing Aids and DNN-based speech enhancement in complex acoustic scenes
IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA23)
by Enric Gusó, Joanna Luberadzka, Martí Baig, Umut Sayin & Xavier Serra
ABSTRACT
We investigate the objective performance of five high-end commercially available Hearing Aid (HA) devices compared to DNN-based speech enhancement algorithms in complex acoustic environments. To this end, we measure the HRTFs of a single HA device to synthesize a binaural dataset for training two state-of-the-art causal and non-causal DNN enhancement models. We then generate an evaluation set of realistic speech-in-noise situations using an Ambisonics loudspeaker setup and record with a KU100 dummy head wearing each of the HA devices, both with and without the conventional HA algorithms, applying the DNN enhancers to the latter. We find that the DNN-based enhancement outperforms the HA algorithms in terms of noise suppression and objective intelligibility metrics.
Comparison of synthesized Virtual Sound Environments with validated Hearing Aid experiments
155th Convention of the Audio Engineering Society
by Umut Sayin
ABSTRACT
Real-life situations are hard to replicate in the laboratory and often discarded during hearing aids optimisation, leading to performance inconsistencies and user dissatisfaction. As a solution, the authors propose a tool set to incorporate real-life conditions in the design, test and fitting of hearing aids. This tool set includes a spatial audio simulation framework for generating large number of realistic situations, a machine learning algorithm focused on prominent hearing aids problems trained with the newly generated data, and a low-cost spatial audio solution for audiological clinics for improved fitting of hearing aids. The current article presents the first results of the spatial audio simulation framework compared to a reference scenario and other existent solutions in literature. First findings demonstrate that synthesized impulse responses with arbitrary source directivity combined with using hearing aid head related transfer functions, with spatial upsampling and Ambisonic domain optimizations, to generate simulated binaural audio can be a powerful tool for generating several real-life situations for further hearing aids research.
Sushi with Einstein: Enhancing Hybrid Live Events with LLM-Based Virtual Humans
23rd ACM International Conference on Intelligent Virtual Agents
by Alon Shoa, Ramon Oliva, Mel Slater & Doron Friedman
ABSTRACT
It is becoming increasingly easier to set up multi-user virtual reality sessions, and these can become viable alternatives to video conference in events such as international conferences. Moreover, it is possible to enhance such events with automated virtual humans, who may participate in the discussion. This paper presents the behind-the-scenes work of a panel session titled “Is virtual reality genuine reality?”, which was held during a physical symposium, “XR for the people,” in June 2022. The panel featured a virtual Albert Einstein, based on a large language model (LLM), as a panelist, alongside three international experts having a live conference panel discussion. The VR discussion was broadcast live on stage, and a moderator was able to communicate with both the live audience, the virtual world participants, and the virtual agent (Einstein). We provide lessons learned from the implementation and from the live production, and discuss the potential and pitfalls of using LLM-based virtual humans for multi-user VR in live hybrid events.
Fostering empathy in social Virtual Reality through physiologically based affective haptic feedback
IEEE World Haptics Conference
by Jeanne Hecquard, Justine Saint-Aubert, Ferran Argelaguet, Claudio Pacchierotti, Anatole Lécuyer & Marc Macé
ABSTRACT
We study the promotion of positive social interactions in VR by fostering empathy with other users present in the virtual scene. For this purpose, we propose using affective haptic feedback to reinforce the connection with another user through the direct perception of their physiological state. We developed a virtual meeting scenario where a human user attends a presentation with several virtual agents. Throughout the meeting, the presenting virtual agent faces various difficulties that alter her stress level. The human user directly feels her stress via two physiologically based affective haptic interfaces: a compression belt and a vibrator, simulating the breathing and the heart rate of the presenter, respectively. We conducted a user study that compared the use of such a “sympathetic” haptic rendering vs an “indifferent” one that does not communicate the presenter’s stress status, remaining constant and relaxed at all times. Results are rather contrasted and user-dependent, but they show that sympathetic haptic feedback is globally preferred and can enhance empathy and perceived connection to the presenter. The results promote the use of affective haptics in social VR applications, in which fostering positive relationships plays an important role.
Activation of human visual area V6 during egocentric navigation with and without visual experience
Current Biology
by Aggius-Vella E., Chebat D.R., Maidenbaum S. & Amedi A.
ABSTRACT
V6 is a retinotopic area located in the dorsal visual stream that integrates eye movements with retinal and visuo-motor signals. Despite the known role of V6 in visual motion, it is unknown whether it is involved in navigation and how sensory experiences shape its functional properties. We explored the involvement of V6 in egocentric navigation in sighted and in congenitally blind (CB) participants navigating via an in-house distance-to-sound sensory substitution device (SSD), the EyeCane. We performed two fMRI experiments on two independent datasets. In the first experiment, CB and sighted participants navigated the same mazes. The sighted performed the mazes via vision, while the CB performed them via audition. The CB performed the mazes before and after a training session, using the EyeCane SSD. In the second experiment, a group of sighted participants performed a motor topography task. Our results show that right V6 (rhV6) is selectively involved in egocentric navigation independently of the sensory modality used. Indeed, after training, rhV6 of CB is selectively recruited for auditory navigation, similarly to rhV6 in the sighted. Moreover, we found activation for body movement in area V6, which can putatively contribute to its involvement in egocentric navigation. Taken together, our findings suggest that area rhV6 is a unique hub that transforms spatially relevant sensory information into an egocentric representation for navigation. While vision is clearly the dominant modality, rhV6 is in fact a supramodal area that can develop its selectivity for navigation in the absence of visual experience.
Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device
Frontiers in Neuroscience
by Shira Shvadron, Adi Snir, Amber Maimon, Or Yizhar, Sapir Harel, Keinan Poradosu & Amir Amedi
ABSTRACT
Current advancements in both technology and science allow us to manipulate our sensory modalities in new and unexpected ways. In the present study, we explore the potential of expanding what we perceive through our natural senses by utilizing a visual-to-auditory sensory substitution device (SSD), the EyeMusic, an algorithm that converts images to sound. The EyeMusic was initially developed to allow blind individuals to create a spatial representation of information arriving from a video feed at a slow sampling rate. In this study, we aimed to use the EyeMusic for the blind areas of sighted individuals. We use it in this initial proof-of-concept study to test the ability of sighted subjects to combine visual information with surrounding auditory sonification representing visual information. Participants in this study were tasked with recognizing and adequately placing the stimuli, using sound to represent the areas outside the standard human visual field. As such, the participants were asked to report shapes’ identities as well as their spatial orientation (front/right/back/left), requiring combined visual (90° frontal) and auditory input (the remaining 270°) for the successful performance of the task (content in both vision and audition was presented in a sweeping clockwise motion around the participant). We found that participants were successful at a highly above chance level after a brief 1-h-long session of online training and one on-site training session of an average of 20 min. They could even draw a 2D representation of this image in some cases. Participants could also generalize, recognizing new shapes they were not explicitly trained on. Our findings provide an initial proof of concept indicating that sensory augmentation devices and techniques can potentially be used in combination with natural sensory information in order to expand the natural fields of sensory perception.
Testing geometry and 3D perception in children following vision restoring cataract-removal surgery
Frontiers in Neuroscience
by Amber Maimon, Ophir Netzer, Benedetta Heimler & Amir Amedi
ABSTRACT
As neuroscience and rehabilitative techniques advance, age-old questions concerning the visual experience of those who gain sight after blindness, once thought to be philosophical alone, take center stage and become the target for scientific inquiries. In this study, we employ a battery of visual perception tasks to study the unique experience of a small group of children who have undergone vision-restoring cataract removal surgery as part of the Himalayan Cataract Project. We tested their abilities to perceive in three dimensions (3D) using a binocular rivalry task and the Brock string task, perceive visual illusions, use cross-modal mappings between touch and vision, and spatially group based on geometric cues. Some of the children in this study gained a sense of sight for the first time in their lives, having been born with bilateral congenital cataracts, while others suffered late-onset blindness in one eye alone. This study simultaneously supports yet raises further questions concerning Hubel and Wiesel’s critical periods theory and provides additional insight into Molyneux’s problem, the ability to correlate vision with touch quickly. We suggest that our findings present a relatively unexplored intermediate stage of 3D vision development. Importantly, we spotlight some essential geometrical perception visual abilities that strengthen the idea that spontaneous geometry intuitions arise independently from visual experience (and education), thus replicating and extending previous studies. We incorporate a new model, not previously explored, of testing children with congenital cataract removal surgeries who perform the task via vision. In contrast, previous work has explored these abilities in the congenitally blind via touch. Taken together, our findings provide insight into the development of what is commonly known as the visual system in the visually deprived and highlight the need to further empirically explore an amodal, task-based interpretation of specializations in the development and structure of the brain. Moreover, we propose a novel objective method, based on a simple binocular rivalry task and the Brock string task, for determining congenital (early) vs. late blindness where medical history and records are partial or lacking (e.g., as is often the case in cataract removal cases).
Inceptor: An Open Source Tool for Automated Creation of 3D Social Scenarios
2023 IEEE Conference on Virtual Reality and 3D User Interfaces
by Dan Pollak, Jonathan Giron & Doron Friedman
ABSTRACT
Inceptor is a tool designed for non-expert users to develop social VR scenarios that includes virtual humans. The tool uses a text based interface and natural language processing models as input, and generates complete 3D/VR Unity scenarios as output. The tool is currently based on the Rocketbox asset library. We release the tool as an open source project in order to empower the extended reality research community.
Breathing based immersive interactions for enhanced agency and body awareness: a claustrophobia motivated study
ACM CHI Conference on Human Factors in Computing Systems
by Iddo Yehoshua Wald, Amber Maimon, Lucas Keniger de Andrade Gensas, Noémi Guiot, Meshi Ben Oz, Benjamin W. Corn MD & Amir Amedi
ABSTRACT
This work explores utilizing representations of one’s physiological breath (embreathment) in immersive experiences, for enhancing presence and body awareness. Particularly, embreathment is proposed for reducing claustrophobia and associated negative cognitions such as feelings of restriction, loss of agency, and sense of sufocation, by enhancing agency and interoception in circumstances where one’s ability to act is restricted. The informed design process of an experience designed for this purpose is presented, alongside an experiment employing the experience, evaluating embodiment, presence, and interoception. The results indicate that embreathment leads to signifcantly greater levels of embodiment and presence than either an entrainment or control condition. In addition, a modest trend was observed in a heartbeat detection task implying better interoception in the intervention conditions than the control. These fndings support the initial assumptions regarding presence and body awareness, paving the way for further evaluation with individuals and situations related to the claustrophobia use case.
Persuasive Vibrations: Effects of Speech-Based Vibrations on Persuasion, Leadership, and Co-Presence During Verbal Communication in VR
IEEE Conference on Virtual Reality and 3D User Interfaces
by Justine Saint-Aubert, Ferran Argelaguet, Marc J.-M. Macé, Claudio Pacchierotti, Amir Amedi, & Anatole Lécuyer
ABSTRACT
In Virtual Reality (VR), a growing number of applications involve verbal communications with avatars, such as for teleconference, entertainment, virtual training, social networks, etc. In this context, our paper aims to investigate how tactile feedback consisting in vibrations synchronized with speech could influence aspects related to VR social interactions such as persuasion, co-presence and leadership. We conducted two experiments where participants embody a first-person avatar attending a virtual meeting in immersive VR. In the first experiment, participants were listening to two speaking virtual agents and the speech of one agent was augmented with vibrotactile feedback. Interestingly, the results show that such vibrotactile feedback could significantly improve the perceived co-presence but also the persuasiveness and leadership of the haptically-augmented agent. In the second experiment, the participants were asked to speak to two agents, and their own speech was augmented or not with vibrotactile feedback. The results show that vibrotactile feedback had again a positive effect on co-presence, and that participants perceive their speech as more persuasive in presence of haptic feedback. Taken together, our results demonstrate the strong potential of haptic feedback for supporting social interactions in VR, and pave the way to novel usages of vibrations in a wide range of applications in which verbal communication plays a prominent role.
The Topo-Speech sensory substitution system as a method of conveying spatial information to the blind and vision impaired
Frontiers in Human Neuroscience
by Amber Maimon, Iddo Yehoshua Wald, Meshi Ben Oz, Sophie Codron, Ophir Netzer, Benedetta Heimler, & Amir Amedi
ABSTRACT
Humans, like most animals, integrate sensory input in the brain from different sensory modalities. Yet humans are distinct in their ability to grasp symbolic input, which is interpreted into a cognitive mental representation of the world. This representation merges with external sensory input, providing modality integration of a different sort. This study evaluates the TopoSpeech algorithm in the blind and visually impaired. The system provides spatial information about the external world by applying sensory substitution alongside symbolic representations in a manner that corresponds with the unique way our brains acquire and process information. This is done by conveying spatial information, customarily acquired through vision, through the auditory channel, in a combination of sensory (auditory) features and symbolic language (named/spoken) features. The Topo-Speech sweeps the visual scene or image and represents objects’ identity by employing naming in a spoken word and simultaneously conveying the objects’ location by mapping the x-axis of the visual scene or image to the time it is announced and the y-axis by mapping the location to the pitch of the voice. This proof of concept study primarily explores the practical applicability of this approach in 22 visually impaired and blind individuals. The findings showed that individuals from both populations could effectively interpret and use the algorithm after a single training session. The blind showed an accuracy of 74.45%, while the visually impaired had an average accuracy of 72.74%. These results are comparable to those of the sighted, as shown in previous research, with all participants above chance level. As such, we demonstrate practically how aspects of spatial information can be transmitted through non-visual channels. To complement the findings, we weigh in on debates concerning models of spatial knowledge (the persistent, cumulative, or convergent models) and the capacity for spatial representation in the blind. We suggest the present study’s findings support the convergence model and the scenario that posits the blind are capable of some aspects of spatial representation as depicted by the algorithm comparable to those of the sighted. Finally, we present possible future developments, implementations, and use cases for the system as an aid for the blind and visually impaired.
Multi-sensory display of self-avatar's physiological state: virtual breathing and heart beating can increase sensation of effort in VR
IEEE Transactions on Visualization and Computer Graphics
by Yann Moullec, Justine Saint-Aubert, Julien Manson, Melanie Cogne, & Anatole Lecuyer
ABSTRACT
In this paper we explore the multi-sensory display of self-avatars’ physiological state in Virtual Reality (VR), as a means to enhance the connection between the users and their avatar. Our approach consists in designing and combining a coherent set of visual, auditory and haptic cues to represent the avatar’s cardiac and respiratory activity. These sensory cues are modulated depending on the avatar’s simulated physical exertion. We notably introduce a novel haptic technique to represent respiratory activity using a compression belt simulating abdominal movements that occur during a breathing cycle. A series of experiments was conducted to evaluate the influence of our multi-sensory rendering techniques on various aspects of the VR user experience, including the sense of virtual embodiment and the sensation of effort during a walking simulation. A first study (N=30) that focused on displaying cardiac activity showed that combining sensory modalities significantly enhances the sensation of effort. A second study (N=20) that focused on respiratory activity showed that combining sensory modalities significantly enhances the sensation of effort as well as two sub-components of the sense of embodiment. Interestingly, the user’s actual breathing tended to synchronize with the simulated breathing, especially with the multi-sensory and haptic displays. A third study (N=18) that focused on the combination of cardiac and respiratory activity showed that combining both rendering techniques significantly enhances the sensation of effort. Taken together, our results promote the use of our novel breathing display technique and multi-sensory rendering of physiological parameters in VR applications where effort sensations are prominent, such as for rehabilitation, sport training, or exergames.
A case study in phenomenology of visual experience with retinal prosthesis versus visual-to-auditory sensory substitution
Neuropsychologia
by Amber Maimon, Or Yizhar, Galit Buchs, Benedetta Heimler, & Amir Amedi
ABSTRACT
The phenomenology of the blind has provided an age-old, unparalleled means of exploring the enigmatic link between the brain and mind. This paper delves into the unique phenomenological experience of a man who became blind in adulthood. He subsequently underwent both an Argus II retinal prosthesis implant and training, and extensive training on the EyeMusic visual to auditory sensory substitution device (SSD), thereby becoming the first reported case to date of dual proficiency with both devices. He offers a firsthand account into what he considers the great potential of combining sensory substitution devices with visual prostheses as part of a complete visual restoration protocol. While the Argus II retinal prosthesis alone provided him with immediate visual percepts by way of electrically stimulated phosphenes elicited by the device, the EyeMusic SSD requires extensive training from the onset. Yet following the extensive training program with the EyeMusic sensory substitution device, our subject reports that the sensory substitution device allowed him to experience a richer, more complex perceptual experience, that felt more “second nature” to him, while the Argus II prosthesis (which also requires training) did not allow him to achieve the same levels of automaticity and transparency. Following long-term use of the EyeMusic SSD, our subject reported that visual percepts representing mainly, but not limited to, colors portrayed by the EyeMusic SSD are elicited in association with auditory stimuli, indicating the acquisition of a high level of automaticity. Finally, the case study indicates an additive benefit to the combination of both devices on the user’s subjective phenomenological visual experience.
Congenitally blind adults can learn to identify face-shapes via auditory sensory substitution and successfully generalize some of the learned features
Scientific reports, 2022
by Roni Arbel; Benedetta Heimler; Amir Amedi
ABSTRACT
Unlike sighted individuals, congenitally blind individuals have little to no experience with face shapes. Instead, they rely on non-shape cues, such as voices, to perform character identification. The extent to which face-shape perception can be learned in adulthood via a different sensory modality (i.e., not vision) remains poorly explored. We used a visual-to-auditory Sensory Substitution Device (SSD) that enables conversion of visual images to the auditory modality while preserving their visual characteristics. Expert SSD users were systematically taught to identify cartoon faces via audition. Following a tailored training program lasting ~ 12 h, congenitally blind participants successfully identified six trained faces with high accuracy.
Furthermore, they effectively generalized their identification to the untrained, inverted orientation of the learned faces. Finally, after completing the extensive 12-h training program, participants learned six new faces within 2 additional hours of training, suggesting internalization of face-identification processes. Our results document for the first time that facial features can be processed through audition, even in the absence of visual experience across the lifespan. Overall, these findings have important implications for both non-visual object recognition and visual rehabilitation practices and prompt the study of the neural processes underlying auditory face perception in the absence of vision.
Effects of training and using an audio-tactile sensory substitution device on speech-in-noise understanding
Scientific Reports, 2022
by K. Cieśla; T. Wolak; A. Lorens; M. Mentzel; H. Skarżyński; Amir Amedi
ABSTRACT
Understanding speech in background noise is challenging. Wearing face-masks, imposed by the COVID19-pandemics, makes it even harder. We developed a multi-sensory setup, including a sensory substitution device (SSD) that can deliver speech simultaneously through audition and as vibrations on the fingertips. The vibrations correspond to low frequencies extracted from the speech input. We trained two groups of non-native English speakers in understanding distorted speech in noise. After a short session (30–45 min) of repeating sentences, with or without concurrent matching vibrations, we showed comparable mean group improvement of 14–16 dB in Speech Reception Threshold (SRT) in two test conditions, i.e., when the participants were asked to repeat sentences only from hearing and also when matching vibrations on fingertips were present. This is a very strong effect, if one considers that a 10 dB difference corresponds to doubling of the perceived loudness. The number of sentence repetitions needed for both types of training to complete the task was comparable. Meanwhile, the mean group SNR for the audio-tactile training (14.7 ± 8.7) was significantly lower (harder) than for the auditory training (23.9 ± 11.8), which indicates a potential facilitating effect of the added vibrations. In addition, both before and after training most of the participants (70–80%) showed better performance (by mean 4–6 dB) in speech-in-noise understanding when the audio sentences were accompanied with matching vibrations. This is the same magnitude of multisensory benefit that we reported, with no training at all, in our previous study using the same experimental procedures. After training, performance in this test condition was also best in both groups (SRT ~ 2 dB). The least significant effect of both training types was found in the third test condition, i.e. when participants were repeating sentences accompanied with non-matching tactile vibrations and the performance in this condition was also poorest after training. The results indicate that both types of training may remove some level of difficulty in sound perception, which might enable a more proper use of speech inputs delivered via vibrotactile stimulation. We discuss the implications of these novel findings with respect to basic science. In particular, we show that even in adulthood, i.e. long after the classical “critical periods” of development have passed, a new pairing between a certain computation (here, speech processing) and an atypical sensory modality (here, touch) can be established and trained, and that this process can be rapid and intuitive. We further present possible applications of our training program and the SSD for auditory rehabilitation in patients with hearing (and sight) deficits, as well as healthy individuals in suboptimal acoustic situations.