As a Specialist in Video Encoding,
I Aim to Create the Ultimate Video Imaging System Based on Photoelectron Information

Improving encoding efficiency through importance-weighted predictions of pixel values and perturbations to the transform coefficient

—Could you tell us about recent research that has improved encoding efficiency of two-dimensional (2D) images?

I’ve spent many years researching methods for compressing still images and video, i.e., encoding, at NTT’s research laboratories. Four years ago, I assumed my current, and main, position at Hosei University, but I continue to research this field while also serving as a visiting researcher at NTT. As you have probably noticed, opportunities to send and receive massive amounts of video by every means have been rapidly increasing, and the key technology underpinning this capability is video encoding and decoding.

A block diagram showing the flow of video from the sender to the receiver is shown in Fig. 1. The video is encoded on the left (with an encoder) and decoded on the right (with a decoder) via the bitstream in the center. However, the commonly used JPEG^*1 and MPEG^*2 standards only specify the decoding process (yellow area); they do not specify the encoding process. In other words, the encoding process still has plenty of room for improvement; thus, encoding is being extensively researched.

Fig. 1. Process for encoding and decoding video images.

I have also been conducting original research on the forward transform process called Range Extensions (RExt): highlighted in red on the left side of Fig. 1 (encoding). First, I’ll introduce our method for improving encoding efficiency by predicting the importance of each pixel. For image compression, the pixel values of the image region to be encoded are predicted by referring to the pixel values of the surrounding image region that has already been decoded. The decoded pixel values of this image region can also be referenced in subsequent encoding. These technologies, called intra-frame prediction and inter-frame prediction, play an important role in improving encoding efficiency. In this context, since some pixels (whether in still images or videos) are frequently referenced and other pixels are not, I thought it would be more effective to encode the frequently referenced pixels with more priority. I therefore have researched this method for improving encoding efficiency through prioritized encoding.

The process flow of this encoding method—our original idea—is shown in Fig. 2. First, the image is encoded using an existing encoder, and the coding unit (CU) information^*3 is extracted. It is thus possible to identify pixels that are referenced more frequently (indicated with the bright, warm colors in the figure) and those that are not. By allocating more bits to the former pixels, it is possible to improve encoding efficiency.

Fig. 2. Increasing encoding efficiency by considering the importance of each pixel in predictions.

The next research I’ll describe concerns improving encoding efficiency by perturbing (varying) the transform coefficient, which involves improving the forward transform process (highlighted in red in Fig. 1). For image compression, the conversion of the prediction-error signal is just as important as the prediction of pixel values that I mentioned earlier. Adding a slight perturbation to the transform coefficient will cause the encoding performance to fluctuate, sometimes improving and sometimes deteriorating. In other words, by trying several coefficients and adopting the perturbation condition that yields the best encoding performance, it is possible to increase the compression ratio, thus improve encoding efficiency. The initial stages of this research were undertaken with undergraduate students, and when the results were submitted to the Transactions of the Institute of Electronics, Information and Communication Engineers (IEICE), it was accepted after the first round of peer review without any comments or requests for revisions. Although I’ve written many papers, this submission was the first time the paper was accepted on the first try.

—What kind of research is involved in density-based encoding of 3D-point-cloud video?

Each point on the surface of a 3D point cloud consists of six pieces of information: three x, y, and z coordinates and three colors, red, green, and blue. This research focuses on converting videos into point clouds. Even a single frame of a point cloud video contains millions of points, which equates to a huge amount of data. Therefore, when the image is moving, new point clouds are generated one after another at a rate of tens of frames per second, and that process requires an enormous amount of information—from tens of megabits to tens of gigabits per second.

In consideration of the above-described circumstances, efficient compression technology for 3D-point-cloud video is needed, and compression technologies such as MPEG V-PCC (Video-based Point Cloud Compression), G-PCC (Geometry-based PCC), and MPEG V-DMC (Video-based Dynamic Mesh Coding) have been internationally standardized. However, to achieve higher-performance and more-efficient video compression than possible with these technologies, graduates who completed their master’s degree in the spring of 2025 are continuing their research on 3D-point-cloud video-compression technologies alongside their main jobs.

Our current research on increasing the density of 3D-point-cloud information is shown in Fig. 3. Assuming “point cloud 1 exists at time t, point cloud 2 at time t+1, and point cloud 3 at time t+2,” we combine all three clouds while aiming for a certain form at a specific time. For example, imagining a piece of fabric fluttering, as shown in the figure, we combine the shapes of the fabric at the subsequent time points (input point clouds 2 and 3) to match the shape of input point cloud 1 (at the top left) at that time. We then average the combined point cloud. This combination makes the video image denser, removes noise, and creates a clean, dense point cloud by a process known as point-cloud information densification. When we inversely transform that point cloud back to the shapes of the first, second, and third frames, the individual shapes are the same as the input, but a cleaner image than before densification can be output.

Fig. 3. Flow of increasing the density of 3D-point-cloud information.

—You are also researching coordinated encoding of visible-light and near-infrared images, correct?

Cameras that can capture near-infrared images and visible-light images simultaneously have become commercially available. When the two images are compared, we see that their shapes are almost identical, but details such as their brightness differ significantly. Assuming a certain party already has a near-infrared image, we consider a scenario in which the other party wants to obtain a visible-light image of the same subject, and the former party wants to send the visible-light image efficiently. (Although color-image encoding is available, I’ll use a grayscale image to explain hereafter.)

The flow of an experiment on co-coding of visible-light and near-infrared images is shown in Fig. 4. The predicted image in the figure is obtained by first creating the rounding-clipping formula to pre-extract values similar to those of a grayscale image from the near-infrared image then using the formula to transform the actual near-infrared image. The result of the transformation is an image that closely resembles the actual grayscale image. This predicted grayscale image is then subtracted from the real grayscale image, and the residual image is sent to the other party. If we look closely at the residual image, we can see a faint image, but when superimposing the predicted image and residual image, they don’t perfectly match and differ slightly. In fact, the residual-image file size was 4.78% smaller than the file size compressed using a high-performance lossless coding scheme for grayscale images. The other party then performs the same process in reverse, and by combining the residual image with their own near-infrared image, they can obtain the same grayscale image as the sender. In a similar manner to the graduate researching the 3D-point-cloud video encoding, another graduate who completed their master’s program in the spring of 2025 is continuing this research alongside their main job.

Fig. 4. Flow of co-coding of visible-light and near-infrared images.

Creating the ultimate video imaging system that generates images in which the real subject and image are indistinguishable

—Would you tell us about your research vision aimed at imaging, storage, transmission, and display at the photon level?

Since moving to Hosei University four years ago, I have established a new research area called optical sensing, which focuses on physical phenomena such as the inherent fluctuations (irregular flight) of photons as a property of light. In my previous interview (March 2024 issue), I discussed “limit-break sensing,” a technology that can estimate brightness even of brightly lit subjects that would otherwise be overexposed. I’m currently researching technology to correct variations in measurements done with image sensors. Since the characteristics of converting light into pixel values differ slightly from one pixel element to another, they become unusable in situations in which high accuracy is required, thus becoming necessary to correct this instability. In response to this challenge, a research proposal to understand and solve the problem of pixel-value instability by microscopically grasping the photoelectron information within pixel elements was approved by the Japan Society for the Promotion of Science, and we will receive a Grant-in-Aid for Scientific Research starting in fiscal year 2026. We are currently working on this challenge with students who have progressed to graduate school.

Looking ahead, I aim to create the ultimate system that consistently controls imaging, transmission, storage, and display of images at a microscopic level, such as that of photons. I’m exploring whether this system can be implemented using the phenomenon by which photons are received and emitted, which is similar to the workings of mirrors, as a fundamental principle. The imaging systems that we use daily convert light into electrons then store them. Pixel values for each frame are then repeatedly acquired according to the number of stored electrons. The display systems we use daily are supposed to emit light with a brightness corresponding to a given pixel value, but the number of photons that they generate inevitably varies over time. Regarding variations in photon quantity, for example, the true brightness of the object being videoed always involves fluctuations (expressed as a Poisson distribution) during the imaging stage. When displaying the videoed object, the presence of such Poisson-distribution-related fluctuations in brightness is also unavoidable. Due to fluctuations following a broad distribution that becomes double-convolved during the imaging-to-display process, the image that we observe contains noise with twice the variance of the noise introduced during imaging; in other words, degradation in image quality of 3 dB is occurred (invited lecture at International Display Workshops 2023).

It is obvious when the object is stationary, but when the object moves or the lighting changes over time, motion blur occurs and is superimposed as new noise. Typically, if the shutter is released once every few tens of milliseconds, the movement within that time is averaged out, and the image in each frame becomes blurred.

When you take pictures with your smartphone, you’ve probably noticed that the colors in the image on the screen clearly differ from those in the actual image. If they were indistinguishable, we would have created the ultimate imaging system. Let’s consider the image reflected in a mirror. In this situation, each photon of light is reflected individually, so none of the aforementioned imaging noise, motion blur noise, or display noise are superimposed on the reflected image. This mechanism is a major motivation for wanting to apply the phenomenon of mirrors to video imaging systems, and it is a research topic that I want to focus on in the future.

For imaging, devices capable of observing even a single photon (single-photon avalanche diode (SPAD)^*4) appeared six to seven years ago and are becoming increasingly popular among researchers. However, in normal living environments in which countless photons are coming and going, such devices reach their light-receiving limit with each image shot. Therefore, when imaging an object, it is necessary to do so in a dimly lit environment with a very narrow aperture to obtain information about the photons coming from the object. It is therefore still difficult to capture the whole situation.

When a SPAD detects a photon, it is difficult to determine whether just one photon was detected or whether two or more photons were detected simultaneously. It is therefore necessary to make several assumptions to correct for the probability of how many photons were detected and what was the original brightness, and a correction method incorporating these assumptions has been established.

A test chart captured with a SPAD camera possessed by NTT Computer and Data Science Laboratories is shown in Fig. 5. It appears as if photons are appearing in various places, but when 130,000 of these images are accumulated (photographed for approximately 0.7 seconds), a realistic image of the test chart is obtained. I believe that we will enter an era when we can compress, transmit, store, and display this information composed of individual photons. Regarding display systems, I believe that we can generate individual photons using quantum dot^*5 technology, but in terms of transmission and storage, we still face challenges such as how to compress the incoming photon information and how to evaluate the error (distortion) between input and output, and these methods have not yet been established.

Fig. 5. Photon imaging and measurement using a SPAD camera.

This ultimate video imaging system operating at the photon level is an unexplored field that includes many new areas of research, and within that field, I want to focus my research on innovations in imaging systems and data encoding.

—You have received numerous awards for your social and academic contributions to research on image compression, correct?

I believe my research on image compression has two main important missions. The challenges are how to achieve high compression within the framework of standardization and how to achieve high compression by devising and introducing methods that have not been used before. The former aims to make bitstreams, which are currently used in video services and products, smaller and easier to distribute in a way that makes a direct contribution to society. The latter could make a social contribution, such as being adopted into future international standards and widely used worldwide, or an academic contribution, such as creating trends in new functions and methods that lead to higher-performance standards. While researching with the aim of contributing to both challenges, I’ve received the following nine awards from both inside and outside the company since my last interview two years ago.

During that period, I have delivered four invited and keynote lectures, which primarily focused on new themes.

I want to improve myself through my activities while serving as a member of international academic societies and further contribute to raising the overall level of research in Japan

—Would you share your thoughts on your experiences serving in numerous important positions, including serving on the IEEE Fellow Committee?

As a member of academic societies—both domestically and internationally—I have been involved in activities that have enabled me to foster interactions with various researchers. Thanks to these activities, I have had the opportunity to be invited to give lectures at events with the presence of researchers whom I wouldn’t normally interact with. I have thus been able to introduce my research topics to a wider audience, learn about new research ideas, and receive questions from perspectives that I wouldn’t normally consider but can use to improve my research. While these opportunities reduce the time that I can devote to my own research, I believe they are truly meaningful.

As I mentioned in my previous interview, I was appointed to the IEEE Fellow Committee in 2019. Subsequently reappointed, I’m scheduled to serve until December 2026. Unfortunately, the number of Fellows promoted from Japan has been low recently. The IEEE Fellowship program is structured so that promotions occur once a year (on January 1st). While 41 researchers from Japan were promoted in 2002, that number decreased to just 9 in January 2026, which ranks Japan two places behind Taiwan (with a population one-fifth that of Japan). As you probably know, as Japan lags behind China in terms of number of published papers, the decline in Japan’s research capabilities is causing concern. To improve this situation, since becoming an IEEE Fellow, I have created opportunities for many promoted researchers to speak and share their experiences with Japanese researchers. Thanks to these efforts, and the hard work of the Japanese academic community leaders, the number of Japanese members on the IEEE Fellow Committee, which had fallen from five to two in 2025, rose back to five in 2026. I hope that this improvement will enhance Japan’s presence and lead to an increase in the number of Japanese individuals promoted to Fellow status. I want to continue to contribute, however modestly, to these promotional measures.

Since 2024, I have also been serving as the president of the Institute of Image Electronics Engineers of Japan (IIEEJ). Unlike large academic societies, IIEEJ suffers drawbacks, such as difficulty in achieving economies of scale in terms of funding; however, the committee members and general members are close, and the committee is highly agile. I am striving to help the IIEEJ evolve in a similar way to how insect species have successfully survived by evolving to become smaller. Through participating in connections between academic societies, I have had the rare experience of attending and presenting at conferences hosted by the presidents I have met.

In January 2026, I was appointed editor-in-chief of the APSIPA (Asia-Pacific Signal and Information Processing Association) Transactions on Signal and Information Processing. Immediately after assuming the position, I have faced difficulties such as the publisher’s operational structure becoming unstable, thus inconveniencing members. However, I have already processed many submitted papers and believe I have been able to make a certain contribution.

—Would you share your thoughts on nurturing students and future generations as well as your message to young researchers?

Since I am primarily based at the university, I want to dedicate my efforts to nurturing outstanding students. My students are young and somewhat immature, but at the same time, they are full of energy and hard working—qualities I cannot easily muster—so they have promise. As I mentioned earlier, several students who have completed their master’s programs in the research lab and took jobs in companies as technical staff have voluntarily requested to collaborate on research with me in their free time, and we hold regular meetings. I also have the option of welcoming them as working professionals pursuing doctoral degrees, which I believe is also an interesting and socially meaningful approach.

I want to convey the following message to young researchers. The environment around us is changing ever more rapidly, so past research methods may no longer be optimal. For example, reading comprehension in English was once essential for research, but I think its importance will diminish in the future. Broad but shallow knowledge can now be largely supplemented by artificial intelligence (AI). If you hone your expressive abilities in your native language, the day may soon come when AI will do most of the work for you.

While narrow but deep knowledge is important, we now live in a world where it is impossible to predict whether that knowledge will be useful in the future. I have heard stories of students with no prior economics training using AI to write highly sophisticated papers on economics that surprised economics professors. Therefore, the ability to decide your own direction and forge your own path will become increasingly important.

My academic advisor during my university days was a very busy professor nearing retirement, so direct guidance from him was limited. I am therefore grateful now for the freedom he allowed me in pursuing my research independently. While I encourage you to consider advice from seniors, you should not blindly accept it. Always think about the essence of the problem, make the choices you believe are best, and lead a research life without regrets.

That said, I think it is a good idea to participate in academic activities once you reach a certain age. You’ll be asked to do various things, and you’ll receive a lot of help from others. At first, it might feel like a burden. If you find yourself having to do work that you don’t really want to do, try my way of relieving stress: consciously imagine yourself doing something fun after the task is finished!