To view PDF files

You need Adobe Reader 7.0 or later in order to read PDF files on this site.
If Adobe Reader is not installed on your computer, click the button below and go to the download site.

Feature Articles: 2020 Showcase

2020 Town—Developing MACHINAKA Service, a Device Integration Service that Utilizes Artificial Intelligence Technology

Takayoshi Mochizuki, Manabu Norota,
and Kousuke Suga


At NTT Service Evolution Laboratories, we are working to develop device integration services that can accurately understand the situation a person is in and provide guidance to the person in accordance with the situation. We are working with partner companies and NTT operating companies in this effort and are applying the results of joint experiments to develop services for settings such as long-term care facilities, financial institutions, museums, and outdoor parks that exist within cities. We use the term MACHINAKA (in the city) services to refer to such services. In this article, we introduce our current efforts and future prospects.

Keywords: communication robots, multi-modal interaction, device integration service


1. Introduction

Currently, most MACHINAKA (in the city) services are provided with human labor. However, with the progress of artificial intelligence (AI) technology, the need for human intervention is gradually decreasing., Inc., a well-known example of a logistics company in the US, has already automated part of its operations using AI and robots, from management of its logistic services to product delivery [1]. The increased use of robots in place of humans has also been seen in the customer service industry recently with the growth of AI [2, 3, 4]. The communication robot “Pepper” of SoftBank Corporation is a typical example of a robot replacing humans in a MACHINAKA service industry. It is an example of leveraging communication robots that can interact intelligently with humans through words and actions using AI. The use of communication robots is expected to lead to a reduction in the cost of human labor as well as greater effectiveness of advertising presented using such robots.

As mentioned, we are starting to see communication robots interacting with people and replacing humans in the customer service field. However, it can be said that most of these interactions are still very limited and are centered on routine one-to-one dialogues with customers, for example, to explain goods and services at a reception counter or provide guidance at facilities. This limited use is presumably because the main feature of communication robots is interacting with people using voice and gestures.

At the NTT laboratories, however, we believe that it is possible to expand the scope of applications for communication robots beyond one-to-one customer service situations by combining communication robots with a number of other devices and systems to constitute a device integration service. We describe here the advances that can be achieved when a device integration service centered around communication robots is realized.

As previously mentioned, the customer service industry is increasingly using communication robots to provide guidance or explain facilities or services using voice. This is occurring more and more frequently in the hospitality industry such as at hotels and reception centers, as well as in the banking and retail industry to provide information to customers. However, combining communication robots with images or videos shown on a display will greatly increase the clarity of information that can be conveyed. Also, connecting communication robots with health monitoring devices such as blood pressure monitors opens up the possibility of providing health advice to users [5].

It is also possible for moderators of recreational activities to use a communication robot as a support device by connecting it with a karaoke system. Furthermore, the use of communication robots in cooperation with various devices placed inside a facility or outdoors within a target area makes it possible to not only provide information and guidance to customers, but also to improve the convenience of getting around in the city and to increase the number of visitors and facilitate visitor movements as well.

In this way, we believe that we can assist peoples’ understanding and encourage their behavior and thus stimulate numerous activities within the city by linking various devices to communication robots. In the following sections, we introduce R-env®, the key technology to make all this happen.

2. R-env, technology supporting device and service integration

For more than 40 years, NTT has carried out research and development (R&D) on speech and audio processing technologies and natural language processing technology. NTT is also a world leader in image processing technology, knowledge processing technology, and various other media processing technologies. Efforts are underway to achieve a natural dialogue between communication robots and humans by making use of these technologies. To expand the role of communication robots and increase their existential value, we started R&D on R-env in 2015. R-env enables robots to be easily connected with other external devices and systems (Fig. 1).

Fig. 1. Overview of R-env®, a platform for expanding the human potential through integration of devices and robots.

R-env was developed as a part of ongoing research on interaction technology aimed at expanding the possibilities of humans. This technology enables easy connection and integration of various devices such as communication robots. By linking these devices, this technology will make it possible for communication robots to accurately understand the situation in which a person is placed, provide guidance to the person in accordance with the situation, and prompt the person to adopt a specific behavior or level of awareness.

With R-env, anyone can develop a device integration service using graphical user interface (GUI) screens on a browser. This is done by creating a state transition diagram, or in other words, by combining actions specified for each device connected to R-env, and the condition for the transition to the next action. R-env can also be used to easily create a complex device integration service by reusing and operating multiple state transition diagrams in parallel. Also, services developed on the GUI can be instructed to continue on with their operation even after closing the browser by keeping it on hold in its execution state. Furthermore, when a new device is added, it can be easily recognized, registered, and prepared for use in the R-env service by simply transmitting the predefined JSON (JavaScript Object Notation) format using WebSocket before use.

3. Device integration service using R-env in MACHINAKA (trial case)

We have been conducting trials with NTT Group companies and their external partners to extract the needs and identify any technical problems of device integration services that utilize R-env [5, 6]. In particular, at the end of July 2016, we initiated a joint demonstration experiment in collaboration with five major NTT operating companies. The purpose of the experiment is to evaluate the effectiveness and acceptability of device integration services in a variety of fields and industries in the city (Fig. 2) [7]. In the following section, we introduce a trial case staged in the Shinjuku area.

Fig. 2. Examples of device integration services by NTT Group companies.

4. Trial in Shinjuku

The number of tourists visiting Japan from abroad has increased rapidly. This is evident from the increase in the government’s target number of foreign tourists to Japan, which was raised from 20 million per year to 40 million per year by 2020 at a tourism related meeting chaired by Prime Minister Shinzo Abe on March 30, 2016 [8].

The driving force behind this is the dramatic increase in the number of tourists from China, which doubled in 2015 compared to the previous year. Tourists from China now account for the largest number of foreign tourists to Japan and greatly exceed the number from Taiwan, the second-largest group [9]. Their primary purpose for visiting Japan is shopping, as symbolized by the expression baku-gai, or shopping spree.

Popular areas for Chinese tourists, in addition to well-known tourist destinations, are Ginza and Shinjuku, where there are many shops [10]. Places such as Shinjuku Golden Gai and the Kabukicho neighborhood are also favored by foreign tourists in Shinjuku [11]. Therefore, from the viewpoint of regional development, services in Shinjuku are required not only for local residents but also for people coming from outside Shinjuku, including tourists.

We carried out two joint demonstration experiments utilizing a device integration service targeting the Shinjuku area. The first one was a joint experiment conducted by three groups: Takashimaya Company, Ltd., which is engaged in the retail business, NTT Communications, and the NTT laboratories. In the experiment, we used a device integration service to inform and guide customers to event venues using multiple robots placed in a commercial facility (department store) from August 3 to August 14, 2016. In particular, robots were placed in the following locations based on the flow of pedestrian traffic.

1) The location with the largest amount of pedestrian traffic, which was the area near the second floor (2F) entrance leading to JR Shinjuku Station;

2) The 9F escalator landing, where the main target customers (families with children) of the event are likely to gather;

3) In the vicinity of the 11F escalator landing in which the event venue was located.

At each location, a device integration service composed of an external sensor linked to a communication robot provided smooth induction to the venue by arousing customer interest in the event through dialogue with customers and by providing floor guidance of the department store (Fig. 3).

Fig. 3. Service image in Shinjuku Takashimaya.

The second case is an effort targeting the broad outdoors and is not limited to the indoor space of a facility. It is a joint trial being conducted by the Shinjuku branch of the Tokyo Chamber of Commerce and Industry, the Shinjuku Tourism Promotion Association, NTT EAST, and the NTT laboratories. The trial provides a device integration service consisting of robots and digital signage at multiple locations within a radius of approximately 1 km from Shinjuku Station. In particular, communication robots installed in shops and in tourist facilities in the area provide access information about the respective shops and facilities by utilizing the public wireless local area network service Shinjuku Free Wi-Fi provided by Shinjuku ward.

Communication robots are also playing an important role to increase the number of visitors to the area by providing attractive services such as a robot stamp rally that utilizes multiple robots in different places that cooperate with one another. We are in the process of validating how these services help to increase the number of visitors and facilitate visitor movements (Fig. 4).

Fig. 4. Service image in Shinjuku area.

5. Future development

We are promoting trials to apply device integration services to a wide variety of fields and industries in addition to the cases introduced here; some examples of where these services may be beneficial include long-term care facilities and tourist facilities. Furthermore, in parallel with the joint demonstration experiment, we are also working to expand the application of R-env by broadening cooperation and promoting joint events with hardware manufacturers and service developers [12].

We are also working to spread and improve the device integration service by expanding four initiatives:

1) Conducting hands-on events of the latest devices, including robots;

2) Regularly holding a hackathon event targeted for individual developers and service developers with no programming experience;

3) Holding field trials in cooperation with service providers;

4) Conducting business trials.

We aim to create new business opportunities using robots by enhancing and promoting the value of our MACHINAKA service using device integration services, and by improving the performance of the associated element technology based on the verification results obtained from the joint demonstration experiment.


[1] K. Bhasin and P. Clark, “How Amazon Triggered a Robot Arms Race,” Bloomberg, June 2016.
[2] Website of MIZUHO FinTech, News, Pepper Provides New In-branch Services,
[3] The DENTSU HO report issued by Dentsu Inc. on July 29, 2015 (in Japanese).
[4] M. Tai, “New Japanese Hotel Offers Service with a (Robotic) Smile,” NIKKEI Asian Review, Sept. 2015.
[5] Press release issued by NTT on July 28, 2015 (in Japanese).
[6] Press release issued by NTT on November 25, 2015 (in Japanese).
[7] Press Release issued by NTT on July 25, 2016 (in Japanese).
[8] Prime Minister of Japan and His Cabinet, “Summary of Tourism Vision to Support the Future of Japan (proposal)” (in Japanese).
[9] Article issued by D2C Smile on April 6, 2016 (in Japanese).
[10] Article issued by ACTZERO on December 14, 2015 (in Japanese).
[11] Article of NIKKEI TRENDY NET issued on June 17, 2016 (in Japanese).
[12] Press release issued by NTT on February 16, 2016 (in Japanese).
Takayoshi Mochizuki
Senior Research Engineer, Networked Robot & Gadget Project, NTT Service Evolution Laboratories.
He received an M.A. in human sciences from Waseda University, Tokyo, in 2001. He joined NTT in 2001 and moved to NTT Service Evolution Laboratories in 2014. His research interests include human-robot interaction and human-machine interfaces. He is currently engaged in R&D on non-verbal interactions that encourage new behavior and awareness of people by utilizing robots and other devices and AI.
Manabu Norota
Senior Manager, Supply Chain & Service Industry Group, NTT Communications Corporation.
He received a B.A. in economics from Fukushima University in 1996. He joined NTT Communications in 1996. He is currently engaged in sales promotion.
Kousuke Suga
Associate Manager, New Business Development Headquarters, NTT EAST Corporation.
He received a B.A. from Rikkyo University, Tokyo, in 1996. He joined NTT EAST the same year. He is currently engaged in developing new services for elderly people to promote their use of the Internet.