Understanding Human Sensorimotor Information Processing and Developing Haptic Devices to Support People with Visual Impairments

How is sensory information formed?

—Would you tell us about recent research on information processing from visual information to movement generation?

My research consistently aims to elucidate and model the information processing involved in the movement generation from sensory input. In my previous interview for this journal, I introduced the characteristics of the underlying steps that we discovered in the process that starts with input of visual information and ends with initiation of movement. Sensory information, however, comes in many forms, including vision, hearing, touch, and somatosensation. Although many previous studies have examined how specific types of sensory information are processed, I think it is also important to consider the fundamental question: How does sensory information processing develop in the first place? Through the course of evolution, organisms have acquired various sensory organs. Yet even without the long timescale required for evolution, sensory organs and the information processing associated with them continue to change after birth. The formation of sensory information processing is extremely important for understanding the external environment and one’s bodily state and for interacting with that environment. As humans and animals grow, their physical size changes, and their relationship with their environment also changes. These changes naturally lead to the idea that the way information entering through the senses is processed must also change through adaptation and learning.

Not all of the previous research has supported this idea, and some emphasized the statistical properties of input as a factor in the development and regulation of sensory information processing. Considering the facts that I’ve just outlined and on previous research, I have approached this question from my background in motor control: to what extent can we explain sensory information processing by assuming that it is formed and adjusted so as to enable appropriate movements? To test this hypothesis, my research aims to elucidate the mechanism of sensorimotor control analytically and constructively. I’m also exploring the possibility of applying this mechanism to the implementation of information processing for machines (robots) with flexible functions like learning humans in the future.

Let me briefly explain where in the brain the information processing for motor control and visual analysis takes place. In the brain, there are several regions involved in motor control around an area known as the motor cortex, which contains groups of cells that send commands to the musculoskeletal system. Many bodily movements are generated when these motor-related brain regions become active and send commands to the musculoskeletal system via the spinal cord.

By contrast, visual information, which provides information about the external world that is needed for movement, is processed in different regions. Information received through the eyes is sent to the visual cortex of the brain. The question of how visual information is analyzed has been addressed over many years in numerous studies, which have revealed the details of the processing specific to each area of the brain. As shown in Fig. 1, the brain region called the primary visual cortex, which performs initial visual analysis, is located far from the motor-related region. Between these areas, other areas are responsible for higher-order visual information processing and the representation of external space, for example, and multi-stage neural processing occurs between the analysis of initial visual information and information processing involved in motor generation. Thus, visual information processing and motion-generation information processing had often been studied by different researchers specializing in each field. In fact, our previous research mainly focused on information processing related to motor control.

Fig. 1. Motor-related areas and early visual areas of the brain.

Now, I’d like to introduce two slightly older studies. One is an interesting experiment concerning the behavior of kittens [1]. An “active” kitten can freely move within an experimental apparatus, and the visual information surrounding the kitten changes in accordance with its movements. A “passive” kitten is placed in a box and simply gets “carried along” with the active kitten’s movements. Although the visual stimuli experienced by the passive kitten are the same as those experienced by the active kitten, they are not synchronized with the movements of the kittens’ legs. After this experiment, the active kitten could still perform normal movements synchronized with the visual information it receives, whereas the passive kitten placed in the box could not. In other words, this experiment suggests that normal visual information processing (development of vision) requires not only the experience of simply seeing but also active visual experiences in which one moves of one’s own volition in a manner that changes visual input.

Another example is a well-known study [2] published in the journal Nature. This experimental study involved raising cats in striped boxes from before the critical period crucial for development of vision. It revealed that many cells in the visual cortex of cats raised in this environment developed a response to vertical stripes. This finding has been followed up with various verification studies, and the mechanism by which the response characteristics in visual-cortex cells are formed has become clearer. These earlier research findings suggest that voluntary exercise not only trains the motor cortex but also simultaneously trains the processing of sensory information, including vision. Some researchers in visual information processing suggest that visual information processing develops solely through visual experiences, and I believe that this suggestion can sometimes explain certain aspects of visual information processing. However, considering the study that I mentioned earlier, in a similar way to other studies on the sensorimotor system, I want to investigate the extent to which sensory information processing can be explained by the idea that sensory information processing is acquired or modified for the purpose of movement or the information processing necessary for movement.

In my previous interview [3], I discussed two points: how humans possess reflexive, latent information processing by which the hands move when movement occurs in the field of vision; and these characteristics suggest that information processing derived from self-motion appears to play an important role in reaching control. The constructive approach used in these studies is also applied in the research that I’ll introduce now. First, we recorded visual information from a first-person perspective while the observer was moving in various situations. These first-person videos were then input into a convolutional neural network (CNN), which is widely used in artificial intelligence (AI) technology, and the network was trained through deep learning to estimate the corresponding movement of the observer’s head, that is, self-motion. CNNs are one of the technologies that have driven the recent development of AI, but their original form was proposed by Professor Kunihiko Fukushima, who was inspired by the properties and hierarchical structure of cells in the visual cortex. When we carried out this type of learning, we found that the visual analysis properties formed within the CNN closely resembled the latent response properties of the hand. I discussed this point in detail in the previous interview.

Continuing from the topic of CNNs, I’ll now explain a computational experiment that was inspired by the cat experiment that I mentioned earlier. First, an original first-person video of a subject in motion is taken and converted so that only vertical or horizontal stripes are visible. The video images are then used to train a CNN, and the kernel characteristics acquired in the CNN are investigated. In other words, an experiment in which kittens are raised in a striped environment was regarded as a simulation of the process by which they acquire the ability to convert visual information into motor information. As a result, when the CNN was trained on videos containing vertical stripes, many kernels that responded to vertical stripes were formed in the lower layers of the network. When the CNN was trained on videos containing horizontal stripes, the resulting distribution contained many kernels that responded to horizontal stripes (Fig. 2). In other words, simply by training the model to estimate one’s own movement—self-motion, an important state variable for bodily movement—from visual input, we were able to reproduce in the model the formation of specific properties of visual cortical neurons observed during the postnatal critical period, depending on the visual information in the environment. While this characteristic may not necessarily occur solely through self-motor learning, it suggests that representing bodily movements may play a role in the formation of fundamental functions in visual analysis.

Fig. 2. Differences between orientation-selectivity distributions of vertical- and horizontal-striped images used for training, and the kernels in the first layer of a CNN.

Visualizing the activity of the entire brain

—You’re also conducting research to understand the overall brain activity that occurs during movement, correct?

Correct. I’d like to introduce another research project that I’ve been working on; namely, a collaborative project with the Okinawa Institute of Science and Technology (OIST) analyzing the brain activity of marmosets^* during free movement (Fig. 3). The brain-activity data of marmosets was recorded at RIKEN by using numerous electrodes placed over a wide area of the cerebral cortex. Since this measurement can be conducted while the marmosets are in their normal state of everyday activity, it provides data that contains a very large amount of information. We have been provided with some of these data, and our research aims to understand the flow and processing of information in the cerebral cortex by visualizing the changes that occur in different parts of the brain when head movements occur.

Fig. 3. Visualization of brain activity associated with self-motion.

When we look at videos that visualize brain activity in this manner, we can see that parts of the brain begin to become active just before the head starts moving. The motor-related areas become active first, the frontal lobe becomes active next, and as soon as the head starts moving, the visual cortex, the parietal association cortex, and temporal lobe are activated. While the activity of individual brain areas has been studied extensively, it has been difficult to study brain activity related to sensorimotor activity during free movement. Although this research is just beginning, by collaborating with OIST and RIKEN, we hope to advance our understanding of conscious and unconscious sensorimotor information processing that occurs continuously while a person is interacting with the environment during free movement.

—Would you tell us about the prospects for basic research on information processing by the visual-motor system?

We had been focused on researching latent information processing related to quick motor control from vision. Our research on marmosets has revealed new approaches and important clues to the question of how latent information processing is structured in the brain. We believe that if we can understand brain information processing to a certain extent, we may be able to model it constructively and apply it to robotic motion systems. I also have a lab at Waseda University, where many professors are researching robotics, so I want to collaborate with them to advance research on integrating brain information processing into robotics. Recently, the development of high-performance robots in other countries has been promoted on the Internet and has become a hot topic. From my perspective, however, their movements seem a bit unnatural. Compared with human movement, their movements are awkward and clearly different. I believe this difference stems from an insufficient understanding of the latent information processing capabilities that humans possess, which has not yet been implemented in robots. In other words, they lack a built-in mechanism for learning through dynamic interaction with the external environment. I therefore intend to continue my research with the aim of overcoming these challenges and pioneering a new field in AI robotics.

Supporting the actions of visually impaired people through the combination of haptic feedback and visuals

—Would you tell us about the recent progress of Buru-Navi?

Two years ago, at Sight World, an exhibition of equipment used by visually impaired people, we collaborated with NTT DOCOMO in a demonstration of our mobile haptic device called Buru-Navi developed at NTT Communication Science Laboratories. Although many of the visually impaired participants were using this device for the first time, they quickly understood the pulling sensation that it conveys and evaluated it positively. According to recent World Health Organization data, 285-million people worldwide are visually impaired, and this number (including those with weak sight) is expected to continue to increase. As people age, they are more prone to developing cataracts and glaucoma and often suffer complete or partial loss of sight. Visually impaired people currently use white canes to navigate their surroundings. However, despite the provision of tactile paving and other infrastructure, it is not always sufficient, and walking in the street remains difficult.

Our research explores how information and communication devices can make it easier for people with visual impairments to go out freely, regardless of the availability of tactile paving. In 2025, as part of our efforts to further develop Buru-Navi, we collaborated with the Computer Science Institute Co., Ltd. Like us, they are committed to supporting people with visual impairments and have developed a service called Eye Navi.

As a walking-assistance app, Eye Navi uses AI to analyze images obtained from smartphones in real time and provides voice feedback about obstacles, traffic light colors, directions, routes, and other information. However, in some situations, it is difficult to use voice in daily life. First, it can be difficult to convey spatial information using words. For example, if a visually impaired person is told to “At the next intersection, go straight down the second road on the right for 20 meters,” the person won’t be able to recognize “next intersection” or “second road on the right.” In addition, especially outdoors, the sound is often drowned out by noise. Furthermore, ambient sounds are important clues for understanding the state of the environment, and focusing solely on voice as a guide can be dangerous.

In this collaboration, with the cooperation of a completely blind person, we videoed them walking in a real environment. The video clearly showed that visually impaired people face many difficulties in daily life, such as peeling tactile paving, traffic lights that don’t emit sound, the possibility of passing by necessary turns, and various obstacles.

For visually impaired people, having a way to be guided by the hand and be told “this way” is extremely helpful. Considering the points that I mentioned above, we decided to collaborate in creating a system that combines Buru-Navi and Eye Navi in a manner that makes it easier for visually impaired people to walk around a city independently and freely. As part of that collaboration, we demonstrated the Buru-Navi/Eye Navi combined system at the Open House hosted by NTT Communication Science Laboratories in 2025.

As for this combined system, as well as “direction” being conveyed as before, various patterns of vibrations are assigned to inform the situation. This was designed so that, even without words, the system could communicate the user’s current situation through vibration patterns, such as whether they are crossing a pedestrian crossing at a green light, whether they should continue walking straight for a while, or whether they are approaching the next corner (Fig. 4).

Fig. 4. Communication of walking-path status through changes in amplitude pattern.

Since the demonstration was held indoors, the participants could not experience actual walking guidance. However, by projecting first-person videos taken while walking onto a large monitor, we were able to show them the process of having their smartphone camera recognize the videos, then experiencing different patterns of pulling sensations with Buru-Navi. While implementing these features, we also identified challenges in investigating new aspects of human perception and cognition in a manner that demonstrates that this collaboration was meaningful.

—Would you tell us about the future development of Buru-Navi?

As a haptic force-sensation device, Buru-Navi could be applied to a wide range of applications. Regarding the application of walking guidance for visually impaired people, peripheral technologies are still insufficient, thus it has not yet been commercialized. However, rather than giving up at this point, I intend to continue working with students to develop better haptic devices and conduct research aimed at practical implementation of these innovations. Collaborations with other companies and venture development might be better handled by universities because they offer greater agility. In fact, many universities are engaged in ventures based on research results, and since we’ve managed to devise an interesting device, I want to continue asking ourselves whether it will be useful to society. Other than Buru-Navi, several pedestrian-guidance technologies are being developed. I believe it would be best to exchange information with developers and researchers concerned with those technologies and enable visually impaired persons to choose the technology that suits their preferences.

As well as visually impaired people, people with normal vision can get lost during their daily lives, especially in large buildings or underground passages. I believe that a device like Buru-Navi would be extremely useful in such situations, and I hope to develop the technology on which it is based so that many people can use it.

At university, I want to conduct exciting research with imaginative students

—You are currently researching primarily at a university. Have you noticed any changes in your environment compared with when you were employed by a company?

I’m working primarily at a university while serving as a visiting senior distinguished researcher at NTT Communication Science Laboratories. Unlike companies, universities require researchers to secure their own funding; even so, I feel a great deal of freedom in having no restrictions on research topics in a way that enables you to pursue various research projects based on your ideas. Within a single company, the organization’s goals and values are, naturally, fully expressed; in contrast, universities are composed of a diverse range of individual experts with diverse values. The freedom also enables you to connect with different companies. However, the biggest difference that I noticed was the presence of students. Students will soon be joining my research lab, and I’ve already had the opportunity to interact with them through several lectures and individual meetings. When I look at those students, their eyes are shining, and they are full of enthusiasm and curiosity to learn. They have a rich imagination and sometimes say things that are completely unexpected, but I think that kind of unbiased, free thinking is important. Therefore, being able to interact with them daily is good for enriching my own imagination and returning my mindset to fundamental issues. As I mentioned in my previous interview, I hope that students and researchers will enjoy their studies and research as much as possible and approach both with interest. As for me, I want to be someone who can convey the fun of research.

My department at university hosts experts in fields such as new materials, engines, heat, fluids, and microdevices, and they are all good at conveying the appeal of their respective fields. Compared with the currently popular fields such as AI, the field of materials science may seem a little mundane. However, when you hear about the cutting-edge technologies in that field, your intellectual curiosity will be stimulated. I want to convey that kind of excitement myself. After all, we can challenge ourselves when research is fun, and if it is fun, we can keep working to overcome high hurdles.

References

[1]	R. Held and A. Hein, “Movement-produced Stimulation in the Development of Visually Guided Behavior,” J. Comp. Physiol. Psychol., Vol. 56, No. 5, pp. 872–876, 1963. https://doi.org/10.1037/h0040546
[2]	C. Blakemore and G. Cooper, “Development of the Brain Depends on the Visual Environment,” Nature, Vol. 228, pp. 477–478, 1970. https://doi.org/10.1038/228477a0
[3]	H. Gomi, “Elucidating the Relationship between Implicit Quick Manual Reactions and Mechanisms of Sensory-motor Information Processing in the Brain,” NTT Technical Review, Vol. 22, No. 1, pp. 1–6, Jan. 2024. https://doi.org/10.53829/ntr202401fr1