Statistics and Learning Theory
Learning Theory I

Exploring Unique Relevance for Mutual Information based Feature Selection

Shiyu Liu, Mehul Motani

Date & Time

01:00 am – 01:00 am


Mutual Information (MI), a measure from information theory, is widely used in feature selection. Despite its great success, a promising feature property, namely the unique relevance (UR) of a feature, remains unexplored. In this paper, we improve the performance of mutual information based feature selection (MIBFS) by exploring the utility of unique relevance (UR). We provide a theoretical justification for the value of UR and prove that the optimal feature subset must contain all features with UR. Since existing MIBFS follows the criterion of Maximize Relevance with Minimum Redundancy (MRwMR) which ignores UR of features, we augment it to include the objective of boosting unique relevance (BUR). This leads to a new criterion for MIBFS, called MRwMR-BUR. We conduct experiments on six public datasets and the results indicate that MRwMR-BUR consistently outperforms MRwMR when tested with three popular classifiers. We believe this new insight can lead to new optimality bounds and algorithms.


Shiyu Liu

National University of Singapore

Mehul Motani

National University of Singapore

Session Chair

Lizhong Zheng

Massachusetts Institute of Technology