Space-time interest points

Local image features or interest points provide compact and abstract representations of patterns in an image. In this paper, we propose to extend the notion of spatial interest points into the spatio-temporal domain and show how the resulting features ofte

Space-timeInterestPoints

IvanLaptevandTonyLindeberg

ComputationalVisionandActivePerceptionLaboratory(CVAP)

Dept.ofNumericalAnalysisandComputerScience

KTH,SE-10044Stockholm,Sweden

{laptev,tony}@nada.kth.se

Abstract

Localimagefeaturesorinterestpointsprovidecompactandabstractrepresentationsofpatternsinanimage.Inthispaper,weproposetoextendthenotionofspatialinterestpointsintothespatio-temporaldomainandshowhowtheresultingfeaturesoftenre ectinterestingeventsthatcanbeusedforacompactrepresentationofvideodataaswellasforitsinterpretation.

Todetectspatio-temporalevents,webuildontheideaoftheHarrisandF¨orstnerinterestpointoperatorsanddetectlocalstructuresinspace-timewheretheimagevalueshavesigni http://www.77cn.com.cningsuchdescriptors,weclassifyeventsandcon-structvideorepresentationintermsoflabeledspace-timepoints.Fortheproblemofhumanmotionanalysis,weillus-tratehowtheproposedmethodallowsfordetectionofwalk-ingpeopleinsceneswithocclusionsanddynamicback-grounds.

1.Introduction

Analyzingandinterpretingvideoisagrowingtopicincom-putervisionanditsapplications.Videodatacontainsinfor-mationaboutchangesintheenvironmentandishighlyim-portantformanyvisualtasksincludingnavigation,surveil-lanceandvideoindexing.

Traditionalapproachesformotionanalysismainlyin-volvethecomputationofoptic ow[1]orfeaturetracking[28,4].Althoughveryeffectiveformanytasks,bothofthesetechniqueshavelimitations.Optic owapproachesmostlycapture rst-ordermotionandoftenfailwhenthemotionhassuddenchanges.Featuretrackersoftenassumeaconstantappearanceofimagepatchesovertimeandmayhencefailwhenthisappearancechanges,forexample,insituationswhentwoobjectsintheimagemergeorsplit.

The

supportfromtheSwedishResearchCouncilandfromtheRoyal

SwedishAcademyofSciencesaswellastheKnutandAliceWallenbergFoundationisgratefullyacknowledged.

Figure1:Resultofdetectingthestrongestspatio-temporalinterestpointinafootballsequencewithaplayerheadingtheball.Thedetectedeventcorrespondstothehighspatio-temporalvariationoftheimagedataora“space-timecor-ner”asillustratedbythespatio-temporalsliceontheright.Imagestructuresinvideoarenotrestrictedtoconstantvelocityand/orconstantappearanceovertime.Onthecon-trary,manyinterestingeventsinvideoarecharacterizedbystrongvariationsofthedatainboththespatialandthetem-poraldimensions.Asexample,considersceneswithaper-sonenteringaroom,applaudinghandgestures,acarcrashorawatersplash;seealsotheillustrationin gure1.

Moregenerally,pointswithnon-constantmotioncorre-spondtoacceleratinglocalimagestructuresthatmightcor-respondtotheacceleratingobjectsintheworld.Hence,suchpointsmightcontainimportantinformationabouttheforcesthatactintheenvironmentandchangeitsstructure.Inthespatialdomain,pointswithasigni cantlocalvari-ationofimageintensitieshavebeenextensivelyinvestigatedinthepast[9,11,26].Suchimagepointsarefrequentlyde-notedas“interestpoints”andareattractiveduetotheirhighinformationcontents.Highlysuccessfulapplicationsofin-terestpointdetectorshavebeenpresentedforimageindex-ing[25],stereomatching[30,23,29],optic owestimationandtracking[28],andrecognition[20,10].

Inthispaperwedetectinterestpointsinthespatio-temporaldomainandillustratehowtheresultingspace-timefeaturesoftencorrespondtointerestingeventsinvideodata.Todetectspatio-temporalinterestpoints,webuildontheideaoftheHarrisandF¨orstnerinterestpointoperators[11,9]anddescribethedetectionmethodinsection2.Tocaptureeventswithdifferentspatio-temporalextents[32],

Local image features or interest points provide compact and abstract representations of patterns in an image. In this paper, we propose to extend the notion of spatial interest points into the spatio-temporal domain and show how the resulting features ofte

wecomputeinterestpointsinspatio-temporalscale-spaceandselectscalesthatroughlycorrespondtothesizeofthedetectedeventsinspaceandtotheirdurationsintime.

Insection3weshowhowinterestingeventsinvideocanbelearnedandclassi edusingk-meansclusteringandpointdescriptorsde nedbylocalspatio-temporalimagederiva-tives.Insection4weconsidervideorepresentationintermsofclassi edspatio-temporalinterestpointsanddemonstratehowthisrepresentationcanbeef cientforthetaskofvideoregistration.Inparticular,wepresentanapproachforde-tectingwalkingpeopleincomplexsceneswithocclusionsanddynamicbackground.Finally,section5concludesthepaperwiththediscussionofthemethod.

2.Interestpointdetection

2.1.Interestpointsinspatialdomain

TheideaoftheHarrisinterestpointdetectoristodetectlocationsinaspatialimagefspwheretheimagevalueshavesigni cantvariationsscaleofobservationσ2

inbothdirections.Foragiven

,suchinterestpointscanbefoundfromawindowedsecondl

σ2momentmatrixintegratedatscalei

=sσ2l µsp=gsp(·;σ2(Lsp2 i) Lspx)LspxLspy(LxLsp

spy

y

)2

(1)whereLspxandLsp

yareGaussianderivativesde nedas

Lspx(·;σ2l)= x(gsp(·;σ2l) fsp

)Lspy(·;σ2l)= y(gsp(·;σ2l) fsp),

(2)

andwheregspisthespatialGaussiankernel

gsp

(x,y;σ2

)=1

2πσ

2exp( (x2+y2)/2σ2).(3)

Astheeigenvaluesλ1,λ2,(λ1≤λacteristicvariationsoffspinboth2)ofµsprepresentchar-imagedirections,two

signi cantvaluesofλterestpoint.Todetectsuch1,λ2indicatethepresenceofanin-points,HarrisandStephens[11]proposetodetectpositivemaximaofthecornerfunctionHsp=det(µsp) ktrace2(µsp)=λ1λ2 k(λ1+λ2)2.

(4)

2.2.Interestpointsinthespace-time

Theideaofinterestpointsinthespatialdomaincanbeex-tendedintothespatio-temporaldomainbyrequiringtheim-agevaluesinspace-timetohavelargevariationsinboththespatialandthetemporaldimen …… 此处隐藏：27646字，全部文档内容请下载后查看。喜欢就下载吧 ……