Modeling individual and group actions in meetings with layer

Abstract—We address the problem of recognizing sequences of human interaction patterns in meetings, with the goal of structuring them in semantic terms. The investigated patterns are inherently group-based (defined by the inpidual activities of meeting

IEEE TRANSACTIONS ON MULTIMEDIA,VOL.8,NO.3,JUNE2006509 Modeling Inpidual and Group Actions

in Meetings With Layered HMMs

Dong Zhang,Student Member,IEEE,Daniel Gatica-Perez,Member,IEEE,Samy Bengio,Member,IEEE,and

Iain McCowan,Member,IEEE

Abstract—We address the problem of recognizing sequences of human interaction patterns in meetings,with the goal of structuring them in semantic terms.The investigated patterns are inherently group-based(de?ned by the inpidual activities of meeting participants,and their interplay),and multimodal(as captured by cameras and microphones).By de?ning a proper set of inpidual actions,group actions can be modeled as a two-layer process,one that models basic inpidual activities from low-level audio–visual(A V)features,and another one that models the interactions.We propose a two-layer hidden Markov model (HMM)framework that implements such concept in a principled manner,and that has advantages over previous works.First,by decomposing the problem hierarchically,learning is performed on low-dimensional observation spaces,which results in simpler models.Second,our framework is easier to interpret,as both inpidual and group actions have a clear meaning,and thus easier to improve.Third,different HMMs can be used in each layer, to better re?ect the nature of each subproblem.Our framework is general and extensible,and we illustrate it with a set of eight group actions,using a public5-hour meeting corpus.Experiments and comparison with a single-layer HMM baseline system show its validity.

Index Terms—Human interaction recognition,multimodal pro-cessing and multimedia applications,statistical models.

I.I NTRODUCTION

D EVISING computational frameworks to automatically

infer human behavior from sensors constitutes an open problem in many domains.Moving beyond the person-centered paradigm[36],recent work has started to explore multiperson scenarios,where not only inpidual but also group actions or interactions become relevant[1],[11],[14],[31].

One of these domains is meetings.The automatic analysis of meetings has recently attracted attention in a number of ?elds,including audio and speech processing,computer vision, human–computer interaction,and information retrieval[3], [4],[18],[22],[27],[35],[38].Analyzing meetings poses a

Manuscript received June7,2004;revised May20,2005.This work was sup-ported in part by the EU5th FWP IST Project M4(IST-2001-34485),the EU 6th FWP IST Integrated Project AMI(FP6-506811,Publication AMI-114),the EU6th FWP IST PASCAL Network of Excellence(IST-2002-506778),all sup-ported by the Swiss OFES,and by the Swiss NCCR IM2.The associate editor coordinating the review of this manuscript and approving it for publication was Dr.Richard B.Reilly.

D.Zhang,D.Gatica-Perez,and S.Bengio are with IDIAP Research Institute, CH-1920Martigny,Switzerland(e-mail:zhang@idiap.ch;gatica@idiap.ch; bengio@idiap.ch).

I.McCowan was with the IDIAP Research Institute,CH-1920Martigny, Switzerland.He is now with eHealth Research Center,Brisbane,QLD4000, Australia(e-mail:iain.mccowan@csiro.au).

Digital Object Identi?er10.1109/TMM.2006.870735persity of technical challenges,and opens doors to a number of relevant applications.

Group activity plays a key role in meetings[27],[38],and this is documented by a signi?cant amount of work in social psychology[24].Viewed as a whole,a group shares informa-tion,engages in discussions,and makes decisions,proceeding through perse communication phases both in single meetings and during the course of a long-term teamwork[24].Recog-nizing group actions is therefore useful for browsing and re-trieval purposes[22],[38],e.g.,to structure a meeting into a sequence of high-level items.

Interaction in meetings is inherently group-based[24]and multimodal[16].In the?rst place,we can view a meeting as a continuous sequence of mutually exclusive group actions taken from an exhaustive set[7],[22].Each of these group actions involves multiple simultaneous participants,and is thus implic-itly constrained by the actions of the inpiduals.In the second place,as the principal modality in meetings,speech has recently been studied in the context of interaction modeling[7],[13], [39].However,work analyzing the bene?ts of modeling indi-vidual and group actions using multiple modalities has been lim-ited[1],[22],[23],[32],despite the fact that actions in meetings, both at the inpidual(e.g.,note-taking or talking),and at the group level(e.g.dictating)are often de?ned by the joint occur-rence of speci?c audio and visual patterns.

In this paper,we present a two-layer hidden Markov model (HMM)framework for group action recognition in meetings. The fundamental idea is that,by de?ning an adequate set of in-pidual actions,we can decompose the group action recogni-tion problem into two levels,from inpidual to group actions. Both layers use ergodic HMMs or extensions.The goal of the lower layer is to recognize inpidual actions of participants using low-level audio–visual(A V)features.The output of this layer provides the input to the second layer,which models inter-actions.Inpidual actions naturally constitute the link between the low-level A V features and high-level group actions.Simi-larly to continuous automatic speech recognition,we perform group action recognition directly on the data sequence,deriving the segmentation of group actions in the process.Our approach is general,extensible,and brings improvement over previous work,which re?ects on the results obtained on a public meeting corpus,for a set of eight group actions based on multimodal turn-taking patterns.

The paper is organized as follow …… 此处隐藏：60783字，全部文档内容请下载后查看。喜欢就下载吧 ……