“Tsunagaru Bistro” (provisional name) that communicates the “emotions” of people who are far apart

This article introduces the next-generation communication tool "Tsunagaru Bistro (provisional name)" project developed by NOMLAB, an open innovation lab of NOMURA GROUP together with NTT DoCoMo.

This article is a report article of the event "Social Good Week 2021" that introduces the "Social Good" projects of NOMURA GROUP.

* Click here for details on “Social Good”
NOMURA GROUP 's Social Good (Part 1 / Part 2)
* Click here for a list of report articles on “Social Good Week 2021”

“Tsunagaru Bistro” (provisional name) that connects the “emotions” of people in remote locations

“Tsunagaru Bistro” (provisional name)*, which is under development in collaboration with NTT DoCoMo and NOMURA Co., Ltd., is a fusion of real space and digital technology that eliminates the discomfort caused by distance and conveys the emotions of people who are far away. It is a next-generation online communication tool that allows you to We discussed the background and outline of its development, as well as its social good effects.

* “Tsunagaru Bistro” is the name of the prototype, and since it will be a different name when it is commercialized in the future, it is written as (tentative name). In the text below, it will be described as "connected bistro".

Yuki Saotome, Relations Marketing Department, Nomura NOMURA Co., Ltd., Ltd.
Joined in 2018. I belong to the newly established Relations Marketing Department this year, and I am promoting work that leads to the development of new businesses by combining individual client solutions and connecting companies. Participated in this project as an account manager for NTT DoCoMo. I usually work on PR events, exhibitions, and showroom projects.

NOMURA Co., Ltd. Planning Division Mina Mochizuki
Joined in 2017. Involved in the development of corporate PR facilities, exhibitions, events, and service stores in the Planning Division. Participated in this project as a planner.

NOMURA Co., Ltd., NOMLAB, Nomura Kogeisha
Joined in 2018. Belongs to the open innovation lab NOMLAB (Nomura Open innovation LAB) of NOMURA GROUP in the CIC (Contents Integration Center). Participated in this project as a planner.

From conveying information to conveying emotions

Yoshitake
NOMURA Co., Ltd. has a department called the Content Integration Center (newly established in 2021) that thinks about spaces starting from content. As a planner, I belong to an organization called NOMLAB, which handles digital content. This project was developed in collaboration with NTT DoCoMo and NOMLAB. NOMLAB, as a laboratory for "digital innovation x place creation", is working with various people inside and outside the company to take on the challenge of a self-developed business model.
By the way, have you ever had a Zoom drinking party? It was a boom for a while, but it seems that it is not being done much now. At work, meetings using online tools such as Zoom have become commonplace, but when I wondered why it was not so popular at private drinking parties, I realized that existing online tools are not suitable for communicating information. However, I thought that it might not be suitable for conveying emotions, which is essential for human communication. This project started with the idea that it would be nice to have an online tool for communicating emotions. Saotome will give you an overview of the project.

Saotome from the Relations Marketing Department (left) and Mochizuki from the Planning Department (right)

Saotome
The "Tsunagaru Bistro" developed in this project is a remote spatial communication system that enables you to experience communication as if you were in front of the other person by connecting distant spaces online. As a collaborative project with NTT Docomo, we announced a prototype at the docomo Open House 2021 exhibition held in January 2021. Currently, we are proceeding with development while conducting repeated verifications in preparation for commercialization.
Originally, NTT Docomo and NOMURA Co., Ltd. had a relationship of ordering and receiving orders, but within that connection, NTT Docomo's R&D Innovation Headquarters, which is the research and development department, was introduced to NOMURA Co., Ltd., Ltd.'s NOMLAB. , and that was the beginning of the collaboration. Our major goal is to create next-generation communication by combining NTT Docomo's communication system and IoT technology with NOMURA Co., Ltd. 's space design and production technology.

The project started from scratch, so at first we repeated brainstorming on the theme of "What is communication?" "It's hard to convey" has become a problem. As an approach to this issue, we jointly developed the first prototype, "HUMANIC DOME".

The first collaboration project "HUMANIC DOME"

"HUMANIC DOME" is a dome with a diameter of about 3m, and a chair is installed inside. It senses and digitizes the breathing, pulse, posture, etc. of the person sitting there, and converts the values into video signals. It is a device that expresses a person's emotions, such as whether the person is relieved or nervous. As the first prototype of emotion transfer, it was exhibited at the ICC (NTT InterCommunication Center) in November 2019 and at the docomo Open House in January 2020.

This "HUMANIC DOME" evolved into the current "connected bistro". From here, I will pass the baton to Mochizuki.

Develop and verify with the aim of conveying emotions

Mochizuki
When conducting meetings and meetings using online tools, you may have experienced that it is difficult to read the other person's reaction or that it is difficult to convey your passion. We set out to develop a “connected bistro” with the recognition that online communication is not enough to convey “emotions”. We hypothesized that the five senses of humans are important in communication, and proceeded with the production and verification of a prototype, assuming a meal scene where the five senses are most effective.

The “Tsunagaru Bistro” space is likened to a mobile restaurant in a sightseeing train, and by linking the scenery from the train window, people in separate spaces can feel that they are in the same space. In such a space, we created content that allows people to talk and eat as if they were in front of each other.

“Tsunagaru Bistro” differs from conventional online tools in six ways.
The first is being able to make eye contact. With conventional tools, it was impossible to judge who the other person was talking to without listening to what they said. “Tsunagaru Bistro” is designed with a small camera lens embedded in the front so that you can make eye contact with the person you are interacting with.

The second is a moving train motif, and the same scenery is projected on the wall display on the side so that it can be shared. The scenery changes at the same time, creating a feeling of moving together. I think it is possible to divert this tool for business use by changing the image on the display to an office scene or document sharing.

The third is that you can see all the movements of the entire upper body. By projecting the movements and gestures of the other person, you can communicate more closely. This is a big point of differentiation compared to the conventional online stools that only show from the chest to the top.

The fourth is the introduction of a stereophonic sound system. The same sound can be heard from the same direction, realizing a more natural communication space.

The fifth is that you can see the movement of the opponent's hands, which is included in the third, that you can see the movement of the entire upper body. This time, as a bistro, the motif is a meal scene, so it is important to be able to see what the other person is eating and how much the meal is progressing.

Finally, the sixth point is that the five points I just introduced are aligned, and crosstalk and parallel talk are realized. With today's online tools, while one person is speaking, the other person's microphone is muted, making it difficult to read the other person's reaction and hindering smooth communication. In the "Tsunagaru Bistro", you can see the movements and gestures of the other person's hands, and the 3D sound system allows you to have a natural conversation with the person in front of you or diagonally across from you, allowing you to be in the same space with multiple people at the same time. It is a big feature that communication is established like this.

Based on the above, we created a prototype and conducted a demonstration experiment from January to May 2021 at NOMURA Co., Ltd. 's Shinkiba warehouse. In addition to verification by the people involved, we also invited various people outside the company to hold trial sessions, listened to their impressions after the experience, and repeated the work of updating on the spot. We also exhibited at docomo Open House 2021, which was held from February 4th to 7th. This event was held online, and during the online business meeting, we provided an overview of the content and heard from customers about their dissatisfaction with existing online tools and the functions they expected from "Tsunagaru Bistro".
Through these verifications, we were able to confirm that the combination of small problems caused by the communication environment, such as the communication environment becoming stuck and the voice becoming difficult to hear, causes a great deal of stress. Even in “Tsunagaru Bistro,” I felt the need to pursue a natural way of communicating gestures, facial expressions, gazes, and voices. In order to realize natural communication that does not differ from the real world, we would like to proceed with development with an eye on interlocking with NTT Docomo's 5G (fifth generation mobile communication system).

Social Significance of “Tsunagaru Bistro”

Yoshitake
From here, I would like to consider what kind of social significance there is for a “connected bistro”.
Today, online communication is a daily occurrence, and I think everyone is using SNS as a matter of course, but there are many troubles such as flames and slander. Things that rarely happen in real-life communication often happen online because today's online communication is imperfect, information is conveyed through text, but emotions are not conveyed. I think. The background to the flames and slander may be that there is a society affected by the modern disease of expressing selfishly without thinking about the receiving side. If we had the right online tools to communicate our emotions, this might not happen. So, I went back to the question of what communication is and tried to think about it.

I believe that the origin of communication stems from the invention of writing. There are various theories, but letters were created by dynasties to develop civilization. The reason why I made it is that the voice has a short transmission distance and disappears as soon as it is emitted, so there is a limit to the number of people who can convey it. Letters were invented because it was necessary to communicate to many people in order to develop a dynasty. The king's intentions did not require dialogue with the other party, and could be conveyed unilaterally, so it was a self-righteous means.

On the other hand, the invention of letters was also a revolutionary invention that made it possible to convey intentions beyond distance and time. It can be said that humans are the first to have an external memory other than their own brains. However, the amount of information that can be conveyed by text is small, and it was difficult to convey the accompanying emotions, atmosphere, and atmosphere.

What about language, then? Language was not invented, it arose, and came into being through song for the purpose of animal courtship. It is said that humans, due to their large brain capacities, were able to sing songs in various patterns, which eventually became language. The language that was born in the interaction with the other party needed not only voice but also the environment of the place to be shared in real time. Language can be said to be a caring means of thinking about the other person and communicating their intentions.

Language is not written, emotions are not information. As the information society advances, I think that we have given priority only to the efficient transmission of information and have not paid attention to the transmission of emotions. "Tsunagaru Bistro" is a caring online tool that allows you to communicate your emotions over distance. In today's communication environment, the more information you transfer, the more you can share the space you are in in real time, and the more you can convey your emotions, the more compassionate communication you can have with people who are far away. can be achieved.

Soichi Yoshitake from NOMLAB

We believe that "Tsunagaru Bistro" is not a replacement for conventional online tools, but can be used and coexisted depending on the purpose. Exchanging emotions is very important for human beings and produces various social good. For example, there are many situations where we cannot meet people we want to meet due to the current corona crisis, but grandparents and grandchildren who are in nursing homes can use this tool to meet face-to-face and chat while eating. Patients receiving treatment at the hospital and their families and friends, or patients battling the same disease, can communicate with each other with this tool. Patients with intractable diseases across the country can connect with a limited number of specialists and face their illness with peace of mind. Alternatively, students can take various classes in remote islands where there are few teachers. At the restaurant, enjoy the food while listening to stories about the ingredients from the farmers in the production area. I believe that the biggest difference between "Tsunagaru Bistro" and existing online tools is that it expands the possibilities for creating various social significances.

In terms of technology, the volume of video and audio data will be large in order to achieve more realistic communication, so we would like to introduce NTT Docomo's 5G technology and proceed with further development. .
Lastly, I received a message from NTT DoCoMo, with whom I am collaborating, so I will read it for you.

“It has been two years since we started collaborating to realize next-generation communication in 2018. During that time, there have been various changes in social conditions, but the new perspectives created by NOMURA Co., Ltd., Ltd. have given us a lot of stimulation and awareness. Let's continue to create new value by fusing each other's areas of expertise."

I would like to continue to think of new ways of communication by combining NTT DoCoMo's technology and NOMURA Co., Ltd. 's strengths in spaces. I would like to bring this “connected bistro” to the world in a good way.

---

An online tool that enables caring communication with people who are far away. I think that there are more people who sympathize with this feeling in this corona misfortune. If you read the dialogue “ “Next Generation Communication” Seen in Collaboration with NTT DoCoMo”, which talks about the process of developing the first prototype “HUMANIC DOME”, both companies will be able to communicate in the next era even before the corona disaster. You can see how they are searching for a way and developing a prototype. If you are interested, please read it. Please look forward to NOMLAB, which will continue to develop with the theme of expanding spatial experiences. (nomlog editorial department)

Text: Sho Iwasaki/Photo: Yui Yasuda (during the event)