VROOM at the top as Microsoft pairs avatars with telerobots
Microsoft Research Cambridge is spearheading stunning new technology poised to revolutionise video communications.
Virtual Robot Overlay for Online Meetings (VROOM) combines augmented and virtual reality to bring life-sized avatars into the workplace in the form of telepresence robots.
The technology is designed to make a person working remotely in VR and someone working in the office wearing a Microsoft-designed HoloLens AR headset feel like they’re in the same place.
With more families confined to home and thousands of companies, especially in the technology field, home working during the coronavirus outbreak – and planning to continue doing so when restrictions lift – this is not only an innovation for its time but also a dynamic play for the future.
The advance was revealed in a paper by Microsoft Research Cambridge trio Brennan Jones, Priscilla Wong and Sean Rintel and their Microsoft Vancouver colleague Yaying Zhang.
As they explain, telepresence robots allow remote users to freely explore a space they are not in and provide a physical embodiment in that space. However, they lack a compelling representation of the remote user in the local space – hence the invention of VROOM which is a two-way system for exploring how to improve the social experience of robotic telepresence.
The technology would appear to be capable of transforming face2face online dialogue domestically and across a range of business, academic and research environments.
For the local user, an augmented reality interface shows a life-size avatar of the remote user overlaid on a telepresence robot. For the remote user, a head-mounted virtual reality interface presents an immersive 360° view of the local space with mobile autonomy.
The VR system tracks the remote user’s head pose and hand movements, which are applied to the avatar. This allows the local user to see the remote’s head direction and hand gestures and the remote user to identify with the robot as an identifiable embodiment of self.
The authors write: “Video communication has enabled global work and personal life and distributed access to education, healthcare, professional services, and more.
“However, traditional 2D video calling is inherently asymmetrical, constraining users’ abilities to achieve common ground, maintain awareness and control and share experiences. People can certainly work around these constraints, but they will still be physically and spatiality limited.
“There are two notable ways to add physical and spatial experiences into video communication. One is to use telepresence robots – effectively video-chat-on-wheels.
“While not yet widely adopted in domestic contexts, they are increasingly common in the workplace. Another is to use an augmented reality system in which remote and local users wear head mounted devices to see one another as avatars in their respective local spaces.
“These avatars can be cartoon-like or more photorealistic, providing something approaching a parametric representation of the user. Both methods are a step ahead of traditional video communication in terms of physical and spatial mobility, autonomy, and bodily identification. However, they both still have limitations.
“Telepresence robots still lock users into 2D screens showing constrained fields of view (FOVs) from a remote camera, so while they allow for more autonomous mobility, they suffer from the same limitations on conveying remote users’ body language and expressions as traditional video communication.
“In addition, low FOVs can result in reduced task performance. While current technologies limit the fidelity of parametric avatars, AR systems enable free use of arm and hand gestures and spatial bodily arrangements. However, mobile autonomy in a remote location is limited to a current shared meeting instance with another person – an AR avatar cannot roam around a remote location.
“The question, then, is how to combine robotic telepresence with mixed-reality avatars to provide the best of both worlds to each endpoint of a video-call experience.’ Hence the arrival of VROOM and move over ZOOM!”
The authors say that, given the evolution of video communication and improvements in allied technologies such as wireless communications infrastructure, more opportunities to collaborate with people not sharing a physical space are arising.
This is enabling a greater number of remote and distributed work meetings and activities. As a result, such technologies are empowering, providing inclusion and new opportunities to those who would otherwise not have them; e.g., people with disabilities who cannot leave the home, people who live far away and cannot afford to travel, or need to be far away to take care of family members, etc.
A telepresence robot is a remotely-controlled movable robot with a screen, speakers, a microphone, and a camera. Such a robot provides an experience that is akin to ‘video-chat on wheels,’ allowing a user to drive around and interact with people in another space.
Usage of telepresence robots has been studied by researchers in collaborative and social contexts such as museum visits, remotely attending academic conferences, outdoor activities and long distance relationships.
The authors provide an inspirational example of usage. In their own words: “Amy is a design director in a motorcycle manufacturing company. She is located in Seattle but has teammates in Shanghai. With VROOM, she can be autonomously present in the Shanghai studio. She has a virtual ‘key to the door’ of the Shanghai office, engaging with the office on her own timetable and without the need to organise meetings with a particular Shanghai colleague.
“Her colleagues, each wearing a HoloLens, can see her avatar present in any room, moving around the building and if they call to her they can see her look back over her shoulder.
“Amy can ‘walk’ around the studio to check project progress from team to team, hold 1:1s in her manager’s office and engage in ad hoc ‘water cooler’ conversations with people she comes across.
“In a specific design session, one team shows five life size clay maquettes of new motorcycle designs. Amy and her colleagues are all able to move around the room discussing the designs, huddling around each model and pointing at various design elements.
“As people move, Amy is always able to know whether others are looking at her, where others are looking, and can also direct her attention to anything in the room’s context.
“At the end of the session, as everyone exits the room and goes back to their open office space, Amy is able to continue conversations with a couple of colleagues on the move.
“At the end of that days’ visit, Amy docks the robot ready for another user. An hour later Red, who works in the Brisbane office, calls into the robot and can move around like Amy.
“Although the robot itself is identical, all of Robert’s Shanghai colleagues are able to know he is there at a glance because they can see that his life-size avatar is different to Amy’s.”
To expand on VROOM further, the inventors are interested in providing the local and remote users with a shared mixed-reality workspace. That is, both remote and local users would be able to spawn virtual objects (such as 3D models, documents, etc.) that they could pin to locations in the real environment and both users would be able to see and interact with the objects.
The local user would see the objects overlaid on the real environment through the AR headset, while the remote user would see the same objects but in VR. This could be applied in areas like home planning (e.g., discussing furniture choices in an apartment) or remote education/training (e.g., trainer and trainee adding virtual arrows, notes, tags in the training space).
VROOM can also be used in scenarios with multiple local and remote users.
More explorations can be done looking at how multiple users interact with each other, how to have multiple remote users see each other’s avatars, and how to reduce cost (e.g. with cheap remotely-controlled robots, instead of telepresence robots).