Objective Quality Evaluation Model for Videophone Services

1. Introduction

Videophone services over IP (Internet protocol) will become key services in the next generation network (NGN). To provide a high-quality service for users, it is extremely important to design and manage the quality of experience (QoE) appropriately. To do this, it is desirable to develop an objective quality evaluation model that can estimate subjective quality from physical quality parameters of videophone services.

2. Objective quality measurement

Objective quality assessment [1] can be categorized into media-layer objective models, packet-layer objective models, parametric models, and hybrid models from the viewpoint of the input information. To estimate the quality perceived by users, media-layer objective models use media signals [2]–[5], packet-layer objective models use information about IP packets [6]–[9], parametric models use quality parameters [10]–[18], and hybrid models use a combination of media signals, IP packets, and quality parameters. Media-layer objective models are highly correlated with subjective quality and used for benchmarking and management. However, this approach is inconvenient for QoE planning because relationships among media quality and quality parameters are not directly considered. Packet-layer objective models are mainly used for in-service quality management [6]. Parametric models are convenient for QoE planning because they formulate the relationships among subjective quality and quality parameters and can estimate quality using quality parameters instead of using captured media signals and IP packets. They enable QoE planners to help ensure that users will be satisfied with end-to-end transmission performance, which avoids over-engineering. These models incorporate network, application, and terminal equipment parameters of high importance to QoE planners. In this article, we describe a parametric model for estimating videophone quality that can be used for application and/or network planning and monitoring.

3. Framework of the model

The framework of the parametric model is shown in Fig. 1. Its input parameters are video and speech quality parameters that are considered important in QoE planning and monitoring. The model consists of three functions for speech quality estimation, video quality estimation, and multimedia quality integration. The degradation caused by pure delay is considered only in the multimedia quality integration function.

Fig. 1. Framework of parametric model for videophone services.

1) Speech quality estimation function

This function estimates the listening speech quality using speech quality parameters. The E-model, which has been standardized as ITU-T Recommendation G.107 [11], is widely used for speech services including IP telephony. It can estimate the overall communication quality using a combination of quality factors. However, NTT proposed a new parametric model [13], [14] that can achieve better performance than the E-model and is also applicable to wideband IP telephony services. We use this NTT model for the speech quality estimation model.

2) Video quality estimation function

This function estimates viewing video quality using video quality parameters. It has three features: One is to estimate the video quality affected by coding distortion. Specifically, we consider the optimal frame rate that maximizes the video quality at each bit rate. Another feature is to create a packet loss robustness factor that indicates the degree of video quality degradation due to packet loss rate. The third is to change the coefficient tables for estimating the video quality of various implementations of videophone applications because the video quality cannot be estimated based simply on codec information. For the E-model, methodologies for deriving speech quality factors affected by the codec are provided in ITU-T Recommendation G.113, Appendix I.

3) Multimedia quality integration function

This function can be used for estimating the overall quality from the listening speech quality, viewing video quality, and end-to-end delay. It considers the individual media qualities, delay (i.e., absolute audiovisual delay and audiovisual media synchronization), and their interactions. The output of this model is multimedia quality, as shown in Fig. 1.

4. Accuracy of parametric model for videophone services

The accuracy of the speech quality estimation functions is described in [13], [14], so this section describes the accuracy of the video quality estimation and multimedia quality integration functions. Using the optimized video quality estimation function for a codec, we estimated the subjective video qualities, as shown in Fig. 2. The mean opinion score (MOS) was given by a five-grade absolute category rating (ACR) method (excellent, good, fair, poor, or bad) [19]. As shown in Fig. 2, its estimation correlates very well with subjective quality. By changing the table of coefficients calculated in advance for each codec, we found that our model could estimate video quality [17], [18]. Next, we estimated the overall quality using estimated individual media qualities and delays. The results for the multimedia quality integration function are shown in Fig. 3. The accuracy of the function was sufficient for practical use. These functions provide reliable information about what users actually require for videophone services, so this method can be used for effective design, implementation, and management of both interactive audiovisual applications and communication networks.

Fig. 2. Accuracy of video quality estimation model.

Fig. 3. Accuracy of multimedia quality integration model.

5. Quality planning using our model

Application and/or network planning to improve video quality is extremely important to avoid over-engineering and to provide users with services appropriately, as shown in Fig. 4. In this section, we show an application and network planning example. It considers the following questions.

– What is the optimal frame rate Ofr for each bit rate condition?

– What is the minimum bit rate Br_Min to achieve the QoE requirement (e.g., MOS3.5)?

– How much packet loss rate Ppl_{Max i}s acceptable for maintaining that QoE?

Fig. 4. Example of application scenario of QoE planning and monitoring.

When the bit rate was set to 40, 82, 160, and 384 kbps, our model gave Ofr of 15, 20, 30, and 30 frames per second (fps), respectively, as shown in Fig. 5. When a QoE planner requires MOS3.5, our model enables us to find Br_Min=82 kbit/s, as shown in Fig. 5. To achieve the above QoE requirement when the bit rate is set to 160 or 384 kbit/s, our model indicates that the maximum packet loss rate Ppl_Max is about 1.2%, as shown in Fig. 6. When the QoE planner requires MOS2.5 (MOS=2.5 is the threshold that is acceptable to 50% of users), the Ppl_Max values were about 6 and 4%, respectively. That reveals that a higher bit rate does not necessarily lead to a high-quality service. Our model is a powerful tool for reflecting such characteristics in network planning.

Fig. 5. Relationship between frame rate and video quality for each bit rate.

Fig. 6. Relationship between packet loss rate and video quality for each bit rate.

6. Conclusion

Our model based on quality parameters is a promising way of designing and monitoring the quality of videophone services. Experimental results showed that its estimates correlate very well with subjective quality, which represents the user’s perception of a service. This model will make it easy to design and manage the QoE appropriately, which is important when providing a high-quality service for users in the NGN.

References

[1]	A. Takahashi, H. Yoshino, and N. Kitawaki, "Perceptual QoS Assessment Technologies for VoIP, IEEE Commun. Mag., Vol. 42, No. 7, pp. 28&ndash34, July 2004.
[2]	"Perceptual Evaluation of Speech Quality (PESQ): an objective method for end-to-end speech quality assessment of narrowband telephone networks and speech codecs, ITU-T Recommendation P.862, Feb. 2001.
[3]	VQEG. Video Quality Experts Group website, http://www.vqeg.org/
[4]	J. Okamoto, T. Hayashi, A. Takahashi, and T. Kurita, "Verification of objective quality assessment method for arbitrary video sequence, IMQA, Sept. 2005.
[5]	"Requirements for an objective perceptual multimedia quality model, ITU-T Recommendation J.148, 2003.
[6]	T. Hayashi, M. Masuda, T. Tominaga, and K. Yamagishi, "Non-intrusive QoS Monitoring Method for Realtime Telecommunication Services, NTT Technical Review, Vol. 4, No. 4, pp. 35&ndash59, 2006.
[7]	A. Clark, "Modeling the Effects of Burst Packet Loss and Recency on Subjective Voice Quality, IP Telephony Workshop, 2001.
[8]	"Conformance testing for narrowband voice over IP transmission quality assessment models, ITU-T Recommendation P.564, July 2006.
[9]	M. Masuda and T. Hayashi, "Non-intrusive Quality Monitoring Method of VoIP Speech Based on Network Performance Metrics, IEICE Trans. Commun., Vol. E89-B, No. 2, pp. 304&ndash312, Feb. 2006.
[10]	A. Takahashi, A. Kurashima, and H. Yoshino, "Objective Assessment Methodology for Estimating Conversational Quality in VoIP, IEEE Trans. on Speech Audio and Language Processing, Vol. 14, No. 6, 1983&ndash1993, Nov. 2006.
[11]	"The E-model, a computational model for use in transmission planning, ITU-T Recommendation G.107, Mar. 2005.
[12]	A. Takahashi, "Opinion model for estimating conversational quality of VoIP, Proc. of IEEE ICASSP 2004, Vol. III, pp. 1072&ndash1075, May 2004.
[13]	A. Takahashi, A. Kurashima, and H. Yoshino, "Objective Assessment Methodology for Estimating Conversational Quality in VoIP, IEEE Trans. on Speech Audio and Language Processing, Vol. 14, No. 6, 1983&ndash1993, Nov. 2006.
[14]	A. Takahashi, A. Kurashima, and H. Aoki, "Quality assessment of wideband speech communication services, NTT Technical Review, Vol. 4, No. 4, pp. 47&ndash51, 2006.
[15]	K. Yamagishi and T. Hayashi, "Analysis of psychological factors for quality assessment of interactive multimodal service, Electronic Imaging 2005, pp. 130&ndash138, Jan. 2005.
[16]	K. Yamagishi and T. Hayashi, "Opinion Model using Psychological Factors for Interactive Multimodal Services, IEICE Trans. Commun., Vol. E89-B, No. 2, pp. 281&ndash288, Feb. 2006.
[17]	K. Yamagishi and T. Hayashi, "Verification of Video Quality Opinion Model for Videophone Services, 2nd ISCA/DEGA Tutorial & Research Workshop on Perceptual Quality of Systems, pp. 143&ndash148, Sept. 2006.
[18]	K. Yamagishi and T. Hayashi, "Opinion Model for Estimating Video Quality of Videophone Services, IEEE GLOBECOM 2006, QRP-1, Nov. 2006.
[19]	"Subjective video quality assessment methods for multimedia applications, ITU-T Recommendation P.910, Sept. 1999.

↑ TOP