Multivariate Mutual Information for Audio Video Fusion

Abstract

Information fusion is an essential part of distributed wireless sensor networks as well as perceptual user interfaces. Irrelevant and redundant data severely affect the performance of the information fusion framework. The work presented in this thesis addresses the problem of acceptability of data from multiple sources. A uni-modal information-theoretic framework for face recognition is first developed. Performance of the proposed method is compared with existing principal component analysis based face recognition algorithms. The uni-modal framework is extended to develop a multi-modal algorithm. The algorithm uses multivariate mutual information to validate the acceptability of data from two sources. Unlike the preceding algorithms, the framework does not require any preprocessing, such as automatic face recognition, etc. Moreover, it also does not condition any statistical modeling or feature extraction and learning algorithms to extract the maximum information regions. The various cases containing a single speaker as well as a group of speakers are analyzed. The applicability of the algorithm is tested by incorporating it with Haar like features of Voila and Jones for lip activity detection.