2010-Convolutional Learning of Spatio-temporal Features

ConvolutionalLearning

ofSpatio-temporalFeatures

GrahamW.Taylor,RobFergus,YannLeCun,andChristophBreglerCourantInstituteofMathematicalSciences,NewYorkUniversity

NewYork,USA

{gwtaylor,fergus,yann,bregler}@cs.nyu.edu

Abstract.Weaddresstheproblemoflearninggoodfeaturesforunder-

standingvideodata.Weintroduceamodelthatlearnslatentrepresen-

tationsofimagesequencesfrompairsofsuccessiveimages.Theconvolu-

tionalarchitectureofourmodelallowsittoscaletorealisticimagesizes

whilstusingacompactparametrization.InexperimentsontheNORB

dataset,weshowourmodelextractslatent“ ow elds”whichcorrespond

tothetransformationbetweenthepairofinputframes.Wealsouseour

modeltoextractlow-levelmotionfeaturesinamulti-stagearchitecture

foractionrecognition,demonstratingcompetitiveperformanceonboth

theKTHandHollywood2datasets.

Keywords:unsupervisedlearning,restrictedBoltzmannmachines,con-

volutionalnets,optical ow,videoanalysis,activityrecognition

1Introduction

Whilethedominantmethodologyforvisualrecognitionfromimagesandvideoreliesonhand-craftedfeatures,therehasbeenagrowinginterestinmethodsthatlearnlow-levelandmid-levelfeatures,eitherinsupervised[1],unsupervised[2–4],orsemi-supervisedsettings[5].Inrecentyears,feature-learningmethodshavefocusedonlearningmultiplelayersoffeaturehierarchiestoextractincreasinglyabstractrepresentationsateachstage.Thishasbeengenerallydonebycompos-ingmodulesofthesamearchitecturesuchasRestrictedBoltzmannMachines(RBM)[2],autoencoders[3],orvariousformsofencoder-decodernetworks[4,6,7]eachofwhicharetrainedunsupervisedandthereforecantakeadvantageoflargeamountsofunlabeledimagedata.Theresulting“deeparchitectures”arethengloballytraineddiscriminatively,withtheideathatthe rstphaseofunsupervisedfeaturelearninghasprovidedaninitializationthatismuchmoresalientforhigh-leveltasksthantheusualrandominitialization.

Mostoftheabovemethodsdonotexploitthepictorialnatureoftheinput,andhavebeenappliedtorelativelysmallimagepatches(typicallylessthan64¢64pixels),becausetheydonotscalewellwiththesizeoftheinput.Thiscanbeaddressedbyusingaconvolutionalarchitecture[1],whichexploitsthefactthatsalientmotifscanappearanywhereintheimage.ThisideahasbeenrecentlyusedinthecontextofRBMs[8,9].Byemployingsuccessivestagesofweight-sharing

2GrahamW.Taylor,RobFergus,YannLeCun,andChristophBregler

andfeature-pooling,deepconvolutionalarchitecturescanachievestablelatentrepresentationsateachlayer,thatpreservelocality,provideinvariancetosmallvariationsoftheinput,anddrasticallyreducethenumberoffreeparameters.

Todate,mostoftheworkonunsupervisedfeatureextractionhasfocusedonstaticimagesbutlittleattentionhasbeengiventolearningaboutthewaythatimagesfromvideoschangeovertime.Thefewworksthataddresstheprob-lem(e.g.[10,6])aretrainedonisolatedpatches(notconvolutionally),andsu erfromthesamelimitationsasstaticmethods.Inthispaper,weproposeamodelthatcanextractmotion-sensitivefeaturesfrompairsofimages(i.e.neighbour-ingframesofvideo).Thefeaturescancapturebothstaticanddynamiccontent.Ourmodelistrainedconvolutionallywhichenablesittoworkonhigh-resolutionimages.We rstapplyittosyntheticdataandshowthatitlearnstorepresent ow-likefeatureswhenthetypeoftransformationsarerestricted.Wethenuseittoextractusefulfeaturesforhumanactivityrecognitioninamulti-stagearchi-tecturethatachievesstate-of-the-artperformanceontheKTHactionsdataset.ResultsarealsoshownonthechallengingHollywood2actionrecognitiondataset.2Relatedwork

OurworkextendstheGatedRBM(GRBM)modelproposedbyMemisevicandHinton[10].TheGRBMisabletoextractdistributed,domain-speci crepresen-tationsofimagepatchtransformations.Duetoitstensorparameterization,itisnotpracticaltoapplythismodeltopatcheslargerthanaboutpN 32q¢32sincethenumberofparametersgrowsasOpN4q.Therefore,ithasonlybeenappliedtolow-resolutionsyntheticimagesofshiftingpixelsorPCA-reducedsamplesoflow-resolutionvideo.Whilethemodelhasbeenshowntoimprovedigitclassi- cationbylearningthetypesoftransformationstowhichtheclassi ershouldremaininvariant,wearenotawareofisapplicationtoadiscriminativetaskonrealvideo.MemisevicandHintonhaverecentlyproposedafactoredformoftheGRBM[11]thatdrasticallyreducesthenumberoffreeparametersbyreplac-ingthethree-wayweighttensorwiththreelow-rankmatrices.Inthepresentwork,wetakeanalternativeconvolutionalapproachtoscalingupthemodel,whichachievestheadditionalbene toftranslationinvariance.SutskeverandHinton[12]http://www.77cn.com.cningsyntheticvideosofbouncingballs,theytrainedamodelwhichwasthenabletogeneratesimilarvideos,butdidnotapplytheirworktodiscriminativetasks.Thesignalfromthepastonlyprovidesatypeof“temporalbias”tothehiddenvariables,whichisfundamentallydi erentfromourthird-orderRBM,wherepastinputsmodulatetheinteractionsbetweenthecurrentinputandthelatentfeaturerepresentation.

Buildingontherapidlygrowingliteratureonsparseover-completedecompo-sitionsofimagepatches[13],CadieuandOlshausen[6]haveproposedatwo-layerprobabilisticmodelthatlearnscomplexmotionfeaturesfromvideo.Incontrasttoourmodel,theyexplicitlyseparatestaticamplitudeanddynamicphaseatthe rstlayer.Thesecondlayerthenlearnshigh-orderdependenciesamongthephasevariables.Deanetal.[14]haverecentlyproposedlearningspatio-temporal

ConvolutionalLearningofSpatio-temporalFeatures3

descriptorsbyrecursivelyapplyingthefeature-signsparsecodingalgorithm[15]to3Dpatchesofvideosextractedatdetectedinterestpoints.Likeourwork,theirdescriptorsareadaptive,buttheirmethodistrainedatthepatchlevel.

State-of-the-artmethodsforactivityrecognitionuseengineeredmotionandtexturedescriptorsextractedaroundinterestpointsdetectedbyspatio-temporalcornerdetectors.Thedescriptorsarethenvector-quantized,pooledovertimeandspaceintoa“bag”,andfedtoanSVMclassi er.Amongthebestperformingmethodsare1)Laptevetal.’sspatio-temporalinterestpoints(STIP)[16]usedinconjunctionwiththe“HOG/HOF”descriptorthatcomputeshistogramsofspatialgradientsandoptic owaccumulatedinlocalspace-timeneighbourhoods …… 此处隐藏：28985字，全部文档内容请下载后查看。喜欢就下载吧 ……