Positive-unlabeled AUC Maximization under Distribution Shift

1. Introduction

In many real-world binary classification tasks such as intrusion detection, fraud detection, and medical diagnosis, class-imbalance frequently occurs, where positive data are much smaller than negative data [1]. For example, malicious activities and disease cases (positive data) are much smaller than benign ones (negative data). In such problems, classification accuracy is an inappropriate evaluation metric because a classifier that predicts all instances as negative can achieve high accuracy while failing to detect positive instances. Instead, the area under the receiver operating characteristic curve (AUC) is widely used [2]. Since the AUC is the probability that a positive instance is scored higher than a negative one, it can adequately evaluate performance even with class-imbalanced data. Attention has thus been directed towards AUC maximization methods, which enable accurate classifiers to be learned from labeled positive and negative data in imbalanced classification tasks [2].

With current AUC maximization methods, it is usually assumed that the training and test data distributions are identical. However, this assumption rarely holds in real-world applications because data distributions can change over time or across environments, which is known as distribution shift [3, 4]. For example, attack patterns can evolve over time in intrusion detection, and patient characteristics can differ across hospitals in medical diagnosis. Although this problem could be alleviated by continuously collecting labeled data from the test distribution and re-training the classifier, such labeling is often costly and time-consuming. Labeled negative data are often difficult to collect even from the training distribution. In intrusion detection, for example, malicious data (positive data) can be collected from public blocklists, whereas benign data from legitimate users (negative data) are often unavailable due to privacy reasons, and manually identifying reliable negative data from unlabeled data requires a high level of expertise.

To overcome these limitations, my colleagues and I previously proposed a distribution shift adaptation method for maximizing the AUC in the positive and unlabeled (PU) setting [5, 6]. In this setting, we assume that PU data from the training distribution and unlabeled data from the test distribution are available, as illustrated in Fig. 1. Note that the unlabeled data in the test distribution do not overlap with the test data used for evaluation. Our goal is to learn a classifier that maximizes the AUC on the test distribution under distribution shift without using labeled negative data. We consider two practically important types of distribution shifts: positive distribution shift [3], where the positive-conditional density can vary, and covariate shift [4], where the input density can vary, as shown in Fig. 2. These shifts frequently arise in real-world imbalanced classification tasks. Under these shift-type assumptions, we theoretically show that the AUC on the test distribution can be rewritten using only the available data [5, 6]. By maximizing the rewritten AUC, we can obtain a desired classifier that maximizes the AUC on the test distribution. Through experiments across real-world public data­sets, we demonstrate that our method outperforms current AUC maximization methods in distribution-shift settings.

Fig. 1. Illustrations of our problem and current settings. (a) In the current AUC maximization setting, it is assumed that training and test distributions are identical and labeled positive and negative data are available for training. Classifiers that maximize the AUC can thus be easily learned. (b) In our problem setting, it is assumed that training and test distributions are different and labeled negative data are not available. Classifiers that maximize the AUC on the test distribution are thus impossible to learn without assumptions of shift types.

Fig. 2. Two specific distribution shifts tackled with our method.

2. Preliminary: AUC maximization

AUC maximization is briefly explained as follows. Let x ∈ X be an input instance and y ∈ {+1, −1} be its class label, where +1 and −1 indicate positive and negative classes, respectively. Here, p^p(x) = p(x|y = +1) and pⁿ(x) = p(x|y = −1) represent positive and negative-conditional densities, respectively. Let s(x) ∈ ℝ be a score function that outputs the positivity of x. The classifier is defined by s(x) with threshold t: y = sign(s(x) − t), where sign is a sign function. The (differentiable) AUC of s is defined as

where 𝔼 denotes the expectation, and σ is the sigmoid function [2]. The AUC takes a value between zero and one and becomes higher when the scores of positive instances are higher than those of negative instances.

When positive and negative data drawn from p^p(x) and pⁿ(x) are available, the AUC can be computed by approximating each expectation with the sample-mean, and we can directly learn the score function by maximizing it with standard stochastic gradient descent methods. However, when negative data are unavailable, we cannot directly compute the AUC, thus cannot maximize it.

3. Our method

Our problem setting is first described in Section 3.1. Our method under positive distribution shift and covariate shift is then described in Sections 3.2 and 3.3, respectively.

3.1 Problem setting

We are given positive data and unlabeled data X_tr from the training distribution:

where denote the positive and negative-conditional densities in the training distribution, respectively, p_tr(x) is the marginal training density, and is the positive class-prior in the training distribution. We are also given unlabeled data X_te from the test distribution:

where denote the positive and negative-conditional densities in the test distribution, respectively, p_te(x) is the marginal test density, and π_te is the positive class-prior in the test distribution.

We assume that there is a distribution shift between the training and test distributions. Specifically, we consider two types of distribution shifts. The first is positive distribution shift [3], where the negative-conditional density does not change but the positive-conditional density can vary. The second is covariate shift, where the input density changes but the conditional density of the class label given the input remains unchanged [4]. Our goal is to learn score function s(x) that can maximize the AUC on the test distribution under the distribution shift by using X_tr, and X_te.

3.2 Positive distribution shift adaptation

Positive distribution shift is formally defined as

This shift often occurs in real-world imbalanced classification tasks. For example, in intrusion detection, malicious adversaries rapidly change their attacks (positive data) to bypass detection systems, while the benign data (negative data) do not change much. Under this shift assumption, we can rewrite the AUC on the test distribution explained in Section 2 as

where we set and C represents a constant term that does not depend on s(x) [5]. Each expectation in this rewritten AUC can be approximated from given PU data and π_tr can also be estimated from given PU data ( and X_tr). Since coefficient and constant C do not affect the optimization of s(x), we can safely ignore them for training. We can thus learn s(x) that maximizes the AUC on the test distribution by maximizing the rewritten AUC with only the available data.

3.3 Covariate shift adaptation

Covariate shift is formally defined as

p_tr(y|x) = p_te(y|x), p_tr(x) ≠ p_te(x).

This shift often occurs in real-world imbalanced classification tasks. In medical diagnosis, for example, the distribution of patients (input instances) often varies due to the differences in hospitals and measurement instruments, even when the input-output relation does not change. Under this shift assumption, we derive two estimators of the AUC on the test distribution [6]. The first one is represented as

where is an importance weight, which can be estimated from unlabeled data (X_tr and X_te) using off-the-shelf density-ratio estimation methods [7], and C represents a constant term that does not depend on s(x). Similarly, the second one is represented as

Since both estimators are equivalent to the original AUC, a linear combination of them is also equivalent to the AUC:

where β ∈ [0, 1] is a weighting hyperparameter. Similar to positive covariate shift, since coefficient and constant C do not affect optimization of s(x), we can safely ignore them for training. Therefore, we can learn the desired s(x) by maximizing the rewritten AUC with only the available data.

4. Experiments

5. Conclusions

We previously proposed a distribution shift adaptation method for maximizing the AUC that uses PU data in the training distribution and unlabeled data in the test distribution. We consider two important types of distribution shifts with our method: positive distribution shift and covariate shift. Under each shift, we theoretically rewrite the AUC on the test distribution using only available data. By maximizing the rewritten AUC, we can learn accurate classifiers that maximize the test AUC. Experiments demonstrated the effectiveness of our method under distribution-shift settings. A limitation of the method is the need to assume a specific type of distribution shift. To alleviate this limitation, one interesting future direction is to extend the method to settings in which a small amount of labeled positive data is also available in the test distribution, which enables AUC maximization without assuming a specific type of distribution shift.

References

Atsutoshi Kumagai Distinguished Researcher, Innovative Computing Architecture Laboratory, Computer and Data Science Laboratories, NTT, Inc. He received a B.S. in informatics and mathematical science from Kyoto University in 2010, and M.S. in informatics from Kyoto University in 2012. In 2012, he joined NTT and is currently a distinguished researcher of NTT Computer and Data Science Laboratories and NTT Social Informatics Laboratories. His research interests include machine learning, data mining, and cyber security.