You need Adobe Reader 7.0 or later in order to read PDF files on this site.
If Adobe Reader is not installed on your computer, click the button below and go to the download site.

Feature Articles: Platform Technologies for Services that Utilize Lifelogs

Prospects for Using Lifelogs in the Medical Field

Tatsuaki Ito^†, Tatsuya Ishihara, Yukihiro Nakamura, Shin-yo Muto, Masanobu Abe, and Yasuhiko Takagi

Abstract

In this article, we introduce studies being conducted in cooperation with Kyoto University Hospital in relation to new medical and healthcare services that make use of various lifelogs in every-day life as well as medical information obtained within medical facilities.

†	NTT Cyber Solutions Laboratories Yokosuka-shi, 239-0847 Japan

1. Introduction

In the medical field, it is necessary to gather as much personal information as possible about patients in order to achieve high-quality diagnosis and treatment. Until now, the personal information used in diagnosis and treatment has basically been gathered and used only within medical facilities. It consists of clinical records, test results, medical images, and other such information. However, the medical information that can be gathered within a medical facility is very limited. It would seem that there is a large amount of data that would be useful for medical purposes within the voluminous and varied data gathered in daily life, most of which is spent outside medical facilities, but such lifelog data has gone unused in most cases.

2. Use of lifelogs for medical diagnosis and treatment

Since June 2009, we have been conducting various studies in cooperation with the Kyoto University Hospital to assess the possibilities for using such lifelog data, obtained outside medical facilities, for medical purposes. In this article, we introduce two such studies. The first aimed to evaluate how vital data obtained outside medical facilities (weight, blood pressure, number of steps walked, etc.) can be useful in medicine. The second concerned the development of technologies that can automatically gather a variety of lifelog data without placing undue burden on the user.

2.1. Experiments linking medical information and lifelogs

In these experiments, the Maiko-Net Web service, which is operated by Kyoto Association for Cooperative Medicine (a non-profit organization) with participation from the Kyoto University Hospital, was used as a platform for accumulating medical information within a medical facility. Maiko-Net is a regional medical infrastructure service linked to medical facilities in Kyoto prefecture, with functions for sharing medical information among medical facilities and allowing the public (patients receiving treatment at a medical facility) to view some of this information. For example, a Maiko-Net user receiving treatment at Kyoto University Hospital can view the results of all past blood tests (cholesterol levels, etc.) at any time. For these experiments, Maiko-Net was linked to “goo Karada Log” (karada is the Japanese word for body), which is a lifelog service on the goo Web portal oriented to the health field provided by NTT Resonant Inc. This created a mechanism allowing healthcare practitioners and the general public to access medical information gathered in medical facilities and lifelog information gathered outside medical facilities in an easy and secure manner (Fig. 1).

Fig. 1. Test system.

In mid-December 2009, an announcement was issued to Maiko-Net users receiving treatment at Kyoto University Hospital in order to recruit participants for experiments. Over several months beginning at the end of December, the recruited participants recorded lifelogs on goo Karada Log using the new mechanism. They were recommended to record a set of lifelog data consisting of about 30 items such as blood pressure. These items were selected as being correlated to lifestyle diseases, such as diabetes and high blood pressure, which occur frequently in the Maiko-Net membership, on the basis of advice from doctors at Kyoto University Hospital. Participant login information for Maiko-Net was also linked to goo Karada Log for the convenience of participants, so that logging into the Maiko-Net personal page also provided seamless login to the goo Karada Log personal page.

At the end of February 2009, we conducted a survey of participants and we had a meeting with doctors at Kyoto University Hospital to evaluate the potential for using lifelogs for medical treatment. The following is a summary of the responses to the participant survey.

(1) Approximately half of the participants were already recording health information daily before these experiments began (although for most of them this consisted of handwritten records). Moreover, most of the health information they were recording was covered by the lifelog items recommended for these experiments.

(2) In the experiments, almost 20% of the participants recorded health information every day using the constructed system.

(3) Almost all of the participants wanted healthcare practitioners to view the lifelog data.

The main comments received in the meeting with medical doctors were as follows.

(1) They were not particularly concerned that the lifelog information gathered outside medical facilities might be too inaccurate.

(2) Measurements such as body weight and blood pressure fluctuate depending on when they are taken, so lifelog data gathered over several days rather than the single measurements taken during out-patient treatment allows trends to be understood, and this is important.

(3) In addition to numerical lifelog data, free comments such as “felt dizzy” are important.

(4) It would be better if average, abnormal, and target values for each lifelog item could be displayed to patients.

Using the above results as a starting point, we will continue to study ways of making lifelog recording easier for patients and also study methods of linking lifelog services to medical data systems in medical facilities, so that patient lifelogs can be viewed and used easily by medical practitioners.

2.2. Gait visualization technology

In rehabilitation and the treatment of lifestyle diseases, it is important to record conditions and activity related to walking. Medical facilities have large on-site analytical equipment that can record gait in detail, and this has been useful for treatment. However, information recorded in a medical facility in this way is extremely limited in time and space, so a way of recording walking behavior under normal conditions is desirable for rehabilitation and other types of treatment. With this in mind, we have developed technology for visualizing lifelog data related to walking in ordinary life. It records when, where, and in what manner a person walks.

In developing this technology, one of our goals is to minimize the load on the user by only requiring the user to carry the data-collection terminal. In gathering continuous lifelog data, it is very important not to require much effort from the user. Considering the proliferation of mobile phones with sophisticated functions such as GPS (global positioning system) and accelerometers and the fact that it is fairly common for people to carry a mobile phone, we decided to use a mobile phone with these functions as the data collection terminal.

This technology has two main features. The first applies signal processing to the acceleration signals to distinguish periods of walking and identify details of the gait. In daily life, we perform various actions, such as resting or riding a train or bus, in addition to walking. To record a representation of this in daily life, the system constantly records acceleration signals, and walking periods are identified by finding segments within these signals that have characteristics that indicate walking. The system can also analyze and display the walking segments in further detail, for aspects like the pace of walking and shifts in balance. This provides a clearer view of what type of walking is occurring.

The second feature creates an easy-to-understand graphical visualization combining time and location information. Walking is greatly influenced by the surrounding environment, so location information can be very meaningful. This feature clearly shows when and where the subject was walking. By including location information, the system records the range of user activity as well as the type of walking.

A schematic of this system is shown in Fig. 2. The user carries a mobile phone with the data-collection application installed at all times. The software collects GPS and accelerometer data periodically and automatically sends it to the server, which accumulates it as a lifelog. The lifelog can be viewed on the server through the Internet by the patient himself, medical staff, or other authorized persons.

Fig. 2. System diagram.

Visualizations of the movement log collected while the same person walked over the same course under two different conditions are shown in Fig. 3:

(1) Normal walking (Fig. 3(a)).

(2) Walking with constraints attached to the right knee and ankle that limited their movement range (Fig. 3(b)).

Fig. 3. Movement log.

The walking course began in the lower right of the map and was approximately 3 km in length. The total change in elevation was about 100 m, with particularly steep inclines toward the end of the course. The marks along the course indicate the person’s walking pace, with the fastest pace shown in red and the color changing to green and blue as the pace decreased. In the normal walking case, the pace was initially fast after the start and slowed as walking continued. In contrast, in the constrained walking case, the pace was not fast after the start and became slower much earlier, as indicated by the change to blue markers.

An example of a detailed analysis of walking at the same location in each case (the middle points in Fig. 3) is shown in Fig. 4. In this example, the smoothness of walking was analyzed by taking the fast Fourier transform of the acceleration signals to compute their frequency components. The graphs show that in the normal walking case there were almost no frequency components other than the walking pace, while the constrained case contained various other frequency components. This result shows that wearing the constraints on the right leg was a hindrance to smooth walking.

Fig. 4. Walking smoothness.

Data can be gathered automatically in this way without placing a burden on the user, and a visualization of normal walking behavior can be created. Compared with conventional measurements of walking made in a medical facility, these measurements cover a much greater extent in both time and space, so this normal activity data can be used to understand patient rehabilitation and recovery conditions. It should also be useful for preventive care and capturing symptoms such as weakening legs or declining activity levels in elderly subjects.

3. Future developments

We intend to develop these lifelog-gathering experiments further by increasing the number of subjects in vital-data experiments and building mechanisms that make it easier to maintain patient motivation for collecting lifelog data. We will also continue to study ways of linking medical information systems at medical facilities to these lifelog services. We plan to conduct experiments on the gait visualization technology in the fields of preventive care and rehabilitation in cooperation with Kyoto University Hospital to study its effectiveness in these areas.

	Tatsuaki Ito Researcher, Network Appliance and Services Project, NTT Cyber Solutions Laboratories. He received the B.E. and M.E. degrees in information engineering from Shizuoka University in 1993 and 1995, respectively. He joined NTT Human Interface Laboratories in 1995. His research interests include lifelog systems and medical informatics.
	Tatsuya Ishihara Researcher, Network Appliance and Services Project, NTT Cyber Solutions Laboratories. He received the B.E. degree in electrical engineering from Waseda University, Tokyo, in 2004 and the M.E. degree in information science and technology from the University of Tokyo in 2006. He joined NTT Cyber Solutions Laboratories in 2006 and is currently studying position estimation of both a user and sensors attached to objects such as home electrical appliances and household furniture in indoor environments. He received the 2009 Young Researcher’s Award from the Institute of Electronics, Information and Communication Engineers (IEICE) of Japan. He is a member of IEICE and the Robotics Society of Japan (RSJ).
	Yukihiro Nakamura Research Engineer, Network Appliance and Services Project, NTT Cyber Solutions Laboratories. He received the B.E. and M.E. degrees from the University of Tsukuba, Ibaraki, in 1992 and 1994, respectively. He joined NTT in 1994. From 2001 to 2005, he was with NTT Communications. He joined NTT Cyber Solutions Laboratories in 2005. His research interests include lifelog systems, network robot platforms, and human-robot interaction.
	Shin-yo Muto Senior Researcher Engineer, Supervisor, Network Appliance and Services Project, NTT Cyber Solutions Laboratories. He received the B.E. and M.E. degrees from Waseda University, Tokyo, in 1988 and 1990, respectively. He joined NTT in 1990. He was also a Visiting Associate Professor at the National Institute of Informatics, Tokyo, during 2000–2003. His research interests include robotics, human-machine interfaces, and lifelog systems. He is a member of the Japan Society of Mechanical Engineers and RSJ.
	Masanobu Abe Professor of the Department of Computer Science, Division of Industrial Innovation Sciences, Graduate School of Natural Science and Technology, Okayama University. He received the B.E., M.E., and Ph.D. degrees in electrical engineering from Waseda University, Tokyo, in 1982, 1984, and 1992, respectively. He joined the Yokosuka Electrical Communications Laboratories of Nippon Telegraph and Telephone Public Corporation (now NTT) in 1984. From 1987 to 1991, he worked at ATR Interpreting Telephony Research Laboratories. In 1989, he joined the Laboratory Computer Science, MIT, USA, as a visiting researcher. From 2007 to 2010, he served as an Executive Manager, Senior Research Engineer, Supervisor of NTT Cyber Solutions Laboratories. He has been a professor at Okayama University since 2010. His research interests include digital speech processing, home networking, consumer electronic appliances, data mining algorithms for lifelogs, and human interfaces. He is a member of the Acoustical Society of Japan (ASJ), IEICE, IEEE, and the Association for Computing Machinery. He received a Paper Award from ASJ in 1996. He is co-author of “Recent Progress in Japanese Speech Synthesis” (Gordon and Breach Science Publishers, 2000).
	Yasuhiko Takagi Manager, R&D Produce Group, NTT Research and Development Planning Department. He received the B.E. degree in engineering science and the Master’s degree in Human and Environmental Studies from Kyoto University in 1993 and 1996, respectively. He joined NTT in 1996. Since then, he has mainly been engaged in medical ICT solution business. He is currently engaged in producing medical and healthcare ICT systems using the results of the NTT research institute.

↑ TOP