Robotic avatar makes you visit Venice from kilometers away

Feeling and moving in a place without being there, this is the main goal of the new iCub robot advanced telexistence system, also called iCub3 avatar system, developed by researchers at IIT-Istituto Italiano di Tecnologia (Italian Institute of Technology) in Genova, Italy. The new system was tested in an online demonstration involving a human operator based in IIT, Genova, and a new version of the humanoid robot iCub, the iCub 3, visiting the Italian Pavilion at the 17th International Architecture Exhibition – La Biennale di Venezia; the two sites are 300km far away and the communication relied on basic optical fiber connection, writes the IIT in a press release.

Researchers demonstrated that the system transports the operator locomotion, manipulation, voice, and face expressions to the robotic avatar, while receiving visual, auditory, haptic and touch feedbacks. This is the first time that a system with all these features is tested using a legged humanoid robot for remote tourism so that the human operator may feel and experience where the avatar is. The system is a prototype and may be further developed to be applied in different scenario, from disaster response to healthcare, and to metaverse too.

Potential in many fields

This result has been obtained by the research team coordinated by Daniele Pucci, Principal Investigator of the Artificial and Mechanical Intelligence (AMI) Lab at IIT in Genova. One of their research goals is to obtain humanoid robots that play the role of avatars, namely a robotic body that acts in place of humans without substituting them but allowing them to be where they cannot.

“We believe that this research direction has a tremendous potential in many fields”, explains Daniele Pucci. “On the one hand, the recent pandemic taught us that advanced telepresence systems might become necessary very quickly across different fields, like healthcare and logistics. On the other hand, avatars may allow people with severe physical disabilities to work and accomplish tasks in the real world via the robotic body. This may be an evolution of rehabilitation and prosthetics technologies.”

Integration with wearable technologies

The system integration was possible thanks to the development of technologies that IIT researchers designed beforehand. Therefore, the iCub3 avatar system is mainly composed by the robot iCub3, which is currently being developed at IIT and represents a new version of the iCub robot, and the wearable technologies – named iFeel –  born in the EU-funded project AnDy and further developed in collaboration with the National Institute for Insurance against Accidents at Work (INAIL) for applications in real cases scenario.

An advanced software architecture, designed by the IIT researchers, controls and manages the interconnection between the iCub3 robot and the iFeel system. Also, this software infrastructure allows the integration of commercial wearable technologies, which complete the iCub3 avatar system. For instance, the remote user walks in-place inside a virtual-reality platform that allows him to move the upper-body freely.

Mimicking the user’s expressions

The iCub3 robot is 25 cm taller than the previous iCub versions, thus measuring 1.25 m and representing a more adequate platform to interact within a human environment. Its balance and locomotion are more robust and able to emulate human movements and physical interaction better. The robot is, therefore, bigger – weighing 52 Kg vs 33 Kg – and has more powerful motors in its legs, thus acquiring a faster locomotion. Moreover, the iCub3 robot differs from the previous platform also for a different actuation mechanics, no longer based on cable-driven joints. On the sensors side, it has an additional depth camera and force sensing of the latest generation withstanding higher robot weight. Lastly, iCub3 has a higher capacity battery, which is located within the torso assembly instead of being included in a rigidly attached backpack.

In the demonstration realized by operating the system from Genova to Venice and backward, the IIT wearable iFeel suite tracks operator’s body motions and the avatar system transfers them onto the iCub3 in Venice, which then moves as the user does in Genova. The user is also provided with a headset that tracks the user expressions, eyelids, and eye motions. These head features are projected onto the avatar, which reproduces them with a high level of fidelity: avatar and human share very similar facial expressions. The user wears sensorized gloves that track his hand motions and, at the same time, provide haptic feedback.

Thanks to the avatar system, the remote user can smile, talk, and shake the hand to the guide in Venice: his avatar smiles, talks, and handshakes accordingly. Analogously, when the guide hugs the avatar in Venice, the operator in Genova feels the hug thanks to the IIT’s iFeel suit that also provides upper body haptics. Moreover, the conversation between the remote user in Genova and the guide in Venice is possible thanks to systems that record and transmit the operator voice so that the avatar can reproduce it in Venice.

Promoting cultural heritage

The transmission was streamed on a standard optical fiber internet connection, resulting in only few milliseconds of delay.

“Our iCub 3 avatar system is validated on a legged humanoid robot allowing remote verbal, non-verbal and physical interaction, which represents a perfect starting point when looking for platforms to emulate humans for all interaction aspects.” – comments Daniele Pucci. “What I also see in our near future is the application of this system to the so-called metaverse, which is actually based on immersive and remote human avatars”.

The Director-General of Contemporary Creativity of the Ministry of Culture and Commissioner of the Italian Pavilion, Onofrio Cutaia, said: “With great pleasure, we received the proposal to collaborate with the Italian Institute of Technology in Genova on this project. We really would like to thank Arch. Alessandro Melis, curator of the Italian Pavilion at the 17th International Architecture Exhibition – La Biennale di Venezia, for sharing our enthusiasm and interacting with iCub 3 in a surprising dialogue between man and robot. A unique opportunity to promote contemporary cultural heritage through new forms of communication. We firmly believe that interdisciplinarity and interaction between languages is the real challenge to be faced, and this is what the Directorate-General will focus on in the coming years”.

Also interesting: Building the metaverse in Europe: open source collaboration platforms gain momentum